Oct 30 20:35:27.871543 ip-10-0-141-115 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:35:28.292840 ip-10-0-141-115 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:35:28.292840 ip-10-0-141-115 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:35:28.292840 ip-10-0-141-115 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:35:28.292840 ip-10-0-141-115 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:35:28.297945 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297867 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.297945 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297945 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.297945 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297948 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297953 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297958 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297961 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297964 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297968 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297971 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297974 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297977 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297980 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297983 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297986 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297989 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297992 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297994 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297997 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.297999 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298002 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298006 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.298040 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298009 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298012 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298015 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298017 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298020 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298022 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298025 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298028 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298030 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298033 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298035 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298038 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298041 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298043 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298046 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298049 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298052 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298055 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298057 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298060 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.298497 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298062 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298065 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298068 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298072 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298075 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298077 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298080 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298082 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298086 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298088 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298091 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298093 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298096 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298099 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298102 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298104 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298107 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298110 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298112 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.298957 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298115 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298118 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298120 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298123 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298125 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298128 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298130 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298133 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298135 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298138 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298141 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298143 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298146 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298148 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298151 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298154 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298156 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298159 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.299420 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298164 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298167 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298170 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298173 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298175 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298178 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298181 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298183 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298186 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298189 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298191 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298194 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298197 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298199 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298202 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298220 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298223 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298225 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298228 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298231 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.299857 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298234 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298239 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298241 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298244 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298247 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.298249 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299026 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299033 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299035 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299038 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299041 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299044 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299047 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299050 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299052 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299055 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299058 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299061 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.300344 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299063 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299067 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299069 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299072 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299075 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299078 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299080 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299083 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299086 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299088 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299091 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299094 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299097 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299099 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299102 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299104 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299107 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299109 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299112 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299114 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.300836 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299117 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299119 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299122 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299125 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299127 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299130 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299133 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299135 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299138 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299140 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299143 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299146 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299148 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299152 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299155 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299157 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299160 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299165 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.301336 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299167 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299170 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299173 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299175 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299178 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299181 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299183 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299186 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299189 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299191 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299194 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299196 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299199 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299202 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299219 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299222 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299225 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299228 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299231 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299233 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.301782 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299237 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299240 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299242 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299245 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299248 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299258 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299263 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299269 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299272 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299275 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299277 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299280 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299283 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299285 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299288 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299290 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299293 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299296 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.302276 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299298 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299301 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299303 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299306 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299309 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299311 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299314 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299317 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299320 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299323 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299326 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299328 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299341 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299345 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299348 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299352 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299840 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299845 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299848 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.302726 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299853 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299856 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299858 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299861 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299864 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299867 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299870 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299872 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299875 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299880 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299886 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299889 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299891 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299894 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299896 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299899 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299902 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299905 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299907 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.303172 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299909 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299913 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299915 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299918 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299920 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299922 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299925 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299928 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299931 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299933 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299936 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299939 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299941 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299944 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299947 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299949 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299952 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299955 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299957 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.303663 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299960 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299962 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299966 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299969 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299972 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299974 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299977 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299979 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299982 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299985 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299987 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299990 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299992 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299995 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.299999 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300002 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300005 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300009 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300011 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.304104 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300014 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300016 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300020 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300022 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300025 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300027 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300032 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300034 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300037 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300040 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300044 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300046 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300049 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300051 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300054 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300058 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300060 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300063 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300065 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300068 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.304572 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300071 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300073 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300076 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300078 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300081 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300083 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300086 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300088 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300091 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300093 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300097 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300099 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300102 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300104 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300107 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300114 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300117 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300120 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300124 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300127 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.305081 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300129 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300132 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300134 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300136 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.300139 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301104 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301121 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301131 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301135 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301140 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301143 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301148 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301152 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301156 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301159 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301162 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301165 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301169 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301172 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301175 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301178 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301181 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301184 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:35:28.305569 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301188 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301192 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301195 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301198 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301202 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301218 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301222 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301226 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301229 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301232 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301235 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301238 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301241 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301244 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301249 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301252 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301257 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301260 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301263 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301266 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301272 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301275 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301277 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301280 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:35:28.306094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301283 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301289 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301292 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301295 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301298 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301301 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301304 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301307 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301310 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301313 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301316 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301319 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301323 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301326 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301330 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301333 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301337 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301340 2577 flags.go:64] FLAG: --help="false" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301343 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301346 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301350 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301353 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301356 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:35:28.306712 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301360 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301363 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301366 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301370 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301373 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301376 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301379 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301381 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301384 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301387 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301390 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301393 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301396 2577 flags.go:64] FLAG: --lock-file="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301399 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301402 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301405 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301416 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301419 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301422 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301425 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301428 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301431 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301434 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301437 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301442 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:35:28.307416 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301446 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301450 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301453 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301456 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301459 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301462 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301465 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301468 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301471 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301478 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301481 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301484 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301487 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301490 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301493 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301496 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301499 2577 flags.go:64] FLAG: --port="10250" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301502 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301505 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-014ffddc146390e0f" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301508 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301511 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301514 2577 flags.go:64] FLAG: --register-node="true" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301516 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301520 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301523 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301526 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:35:28.308109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301528 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301532 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301535 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301539 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301541 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301544 2577 flags.go:64] FLAG: --runonce="false" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301548 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301551 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301554 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301557 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301559 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301562 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301566 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301569 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301573 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301576 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301579 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301582 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301585 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301588 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301591 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301596 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301599 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301602 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301606 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:35:28.309221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301609 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301611 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301614 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301617 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301620 2577 flags.go:64] FLAG: --v="2" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301624 2577 flags.go:64] FLAG: --version="false" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301629 2577 flags.go:64] FLAG: --vmodule="" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301633 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.301636 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301792 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301796 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301800 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301804 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301807 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301810 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301813 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301815 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301818 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301821 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301824 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301828 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.309874 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301832 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301834 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301837 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301839 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301842 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301845 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301847 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301850 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301855 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301857 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301860 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301862 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301865 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301867 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301870 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301872 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301875 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301878 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301880 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.310399 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301882 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301886 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301890 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301894 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301897 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301900 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301904 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301906 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301909 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301912 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301917 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301919 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301922 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301925 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301927 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301930 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301932 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301935 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301938 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301940 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.310884 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301943 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301945 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301948 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301950 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301953 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301955 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301958 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301960 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301963 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301966 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301968 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301971 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301973 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301976 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301978 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301982 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301984 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301987 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301990 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.311442 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301993 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301996 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.301998 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302002 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302005 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302008 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302010 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302013 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302015 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302018 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302021 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302023 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302026 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302028 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302031 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302033 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302036 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302038 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302041 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302043 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.311891 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302046 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302048 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302051 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302053 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302056 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302058 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302061 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302063 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302068 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302070 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302073 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302075 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302078 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302081 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302083 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.302111 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:35:28.312369 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302146 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302152 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302156 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302160 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302163 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302166 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302168 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302171 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302174 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302177 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302179 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302182 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302184 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302187 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302190 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302192 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302195 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302197 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302200 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.312745 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302215 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302218 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302221 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302224 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302227 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302233 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302236 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302239 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302242 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302245 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302247 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302251 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302254 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302258 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302260 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302263 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302265 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302268 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302271 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.313196 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302273 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302277 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302279 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302282 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302285 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302287 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302290 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302293 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302295 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302298 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302301 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302303 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302306 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302308 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302311 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302314 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302316 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302319 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302322 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.313654 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302325 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302328 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302331 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302334 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302337 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302339 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302343 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302346 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302349 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302352 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302354 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302357 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302359 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302362 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302365 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302367 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302370 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302372 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302375 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302377 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.314096 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302379 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302382 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302384 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302387 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302389 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302392 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302395 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302397 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302400 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302403 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302405 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302409 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302411 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302414 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302416 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302419 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302422 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302424 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302427 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302431 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.314590 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302434 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302436 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302439 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302441 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302444 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302446 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302449 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302451 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.302454 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.310832 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.310850 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.311537 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.311545 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311617 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311623 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311627 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311630 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311632 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311635 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.315087 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311638 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311641 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311643 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311646 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311649 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311652 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311656 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311660 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311664 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311667 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311670 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311673 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311676 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311680 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311684 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311687 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311689 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311692 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311695 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.315626 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311698 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311701 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311704 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311707 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311710 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311713 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311715 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311718 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311721 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311723 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311726 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311729 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311731 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311734 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311736 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311739 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311742 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311744 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311747 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311749 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.316068 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311752 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311754 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311757 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311760 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311762 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311764 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311767 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311770 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311773 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311775 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311778 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311780 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311783 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311786 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311791 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311795 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311797 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311800 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311803 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.316551 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311805 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311808 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311810 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311813 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311815 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311818 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311820 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311823 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311826 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311828 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311833 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311836 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311838 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311842 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311845 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311847 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311849 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.317008 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311855 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311857 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311859 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311862 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311865 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311867 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311870 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311873 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311875 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311878 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311883 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311886 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311891 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311894 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311897 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311900 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311902 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.317459 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311905 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311908 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.311911 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.311939 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312075 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312079 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312082 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312085 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312088 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312092 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312095 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312098 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312101 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312103 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312105 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312108 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.317904 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312111 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312114 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312117 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312119 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312122 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312125 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312127 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312130 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312132 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312135 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312138 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312140 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312142 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312145 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312147 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312150 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312152 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312155 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312157 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.318302 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312160 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312162 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312165 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312167 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312169 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312172 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312175 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312178 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312181 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312183 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312186 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312188 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312191 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312193 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312196 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312198 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312202 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312223 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312228 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.318753 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312231 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312234 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312237 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312239 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312242 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312244 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312247 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312249 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312252 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312255 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312258 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312260 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312263 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312266 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312268 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312271 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312274 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312276 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312280 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.319193 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312284 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312287 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312291 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312294 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312297 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312300 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312302 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312305 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312307 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312310 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312312 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312314 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312317 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312320 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312323 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312325 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312328 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312330 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312333 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312336 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.319667 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312338 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312341 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312343 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312346 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312349 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312351 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312354 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312356 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312358 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312361 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312363 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312366 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312368 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312371 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312373 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312377 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.320125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312380 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.312405 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312443 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312448 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312451 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312453 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312456 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312458 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312461 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312464 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312466 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312469 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312471 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312474 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312477 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312479 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:35:28.320561 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312482 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312484 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312487 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312491 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312495 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312498 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312501 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312504 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312506 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312509 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312512 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312514 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312517 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312519 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312522 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312524 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312527 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312531 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312533 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:35:28.320929 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312536 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312538 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312541 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312543 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312546 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312549 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312551 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312554 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312556 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312559 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312562 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312564 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312567 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312569 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312572 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312574 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312578 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312580 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312583 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312586 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:35:28.321392 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312588 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312591 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312593 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312597 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312600 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312602 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312605 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312607 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312610 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312612 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312616 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312618 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312621 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312623 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312626 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312629 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312638 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312642 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312646 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:35:28.321853 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312649 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312653 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312656 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312660 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312664 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312667 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312669 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312672 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312676 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312679 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312681 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312684 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312686 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312689 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312692 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312694 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312697 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312699 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312702 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:35:28.322323 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312705 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312707 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312710 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312713 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312720 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312723 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312726 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312728 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312731 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312734 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312738 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312744 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312748 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:28.312751 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.312804 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.315306 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.316589 2577 server.go:1014] "Starting client certificate rotation" Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.316689 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:35:28.322814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.316731 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:35:28.323298 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.318853 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:35:28.323298 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.322596 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:35:28.332952 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.332935 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:35:28.337814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.337794 2577 log.go:25] "Validated CRI v1 image API" Oct 30 20:35:28.338726 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.338700 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:35:28.342676 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.342656 2577 fs.go:135] Filesystem UUIDs: map[2df800b0-6c15-4695-b98e-453035aabeb1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b6bdb10b-b5f1-481c-a19d-e14546dc99d2:/dev/nvme0n1p3] Oct 30 20:35:28.342753 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.342676 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:35:28.348934 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.347466 2577 manager.go:217] Machine: {Timestamp:2026-10-30 20:35:28.346968676 +0000 UTC m=+0.354830187 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100450 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc278e493b110fdc3f776d45ff989 SystemUUID:ec2dc278-e493-b110-fdc3-f776d45ff989 BootID:f0e261aa-311f-48a3-9cf7-d362ea654d48 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:df:02:73:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:df:02:73:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:79:f9:b1:85:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:35:28.348934 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.347866 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:35:28.348934 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.347998 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:35:28.349480 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.349447 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:35:28.349865 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.349490 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-115.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:35:28.349914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.349870 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:35:28.349914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.349882 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:35:28.349914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.349908 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:35:28.350904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.350894 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:35:28.352274 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.352263 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:35:28.357234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.357220 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:35:28.357371 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.357362 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:35:28.358149 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.358137 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:35:28.358181 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.358151 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:35:28.358181 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.358162 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:35:28.358181 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.358171 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:35:28.358181 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.358179 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:35:28.358510 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.358499 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:35:28.360990 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.360973 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:35:28.366342 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.366327 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:35:28.367221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.367194 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:35:28.367265 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.367239 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:35:28.368770 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368758 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:35:28.368810 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368776 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:35:28.368810 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368782 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:35:28.368810 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368787 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:35:28.368810 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368793 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:35:28.368810 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368799 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:35:28.368810 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368808 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:35:28.368971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368822 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:35:28.368971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368830 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:35:28.368971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368836 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:35:28.368971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368844 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:35:28.368971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.368853 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:35:28.369708 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.369699 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:35:28.369755 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.369710 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:35:28.373738 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.373725 2577 server.go:1263] "Started kubelet" Oct 30 20:35:28.373809 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.373782 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:35:28.373872 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.373822 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:35:28.373912 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.373897 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:35:28.374402 ip-10-0-141-115 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:35:28.375398 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.375378 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:35:28.378108 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.377586 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:35:28.382291 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.382268 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:35:28.383150 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.383133 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:35:28.383769 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.383751 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:35:28.383856 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.383834 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:28.384497 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384464 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:35:28.384497 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384475 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:35:28.384607 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384510 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:35:28.384607 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384588 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:35:28.384607 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384597 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:35:28.384726 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384710 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:35:28.384761 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384729 2577 factory.go:55] Registering systemd factory Oct 30 20:35:28.384761 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384738 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:35:28.384950 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384937 2577 factory.go:153] Registering CRI-O factory Oct 30 20:35:28.384986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384953 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 20:35:28.384986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384967 2577 factory.go:103] Registering Raw factory Oct 30 20:35:28.384986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.384977 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 20:35:28.386090 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.386073 2577 manager.go:319] Starting recovery of all containers Oct 30 20:35:28.398481 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.398458 2577 manager.go:324] Recovery completed Oct 30 20:35:28.399759 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.399739 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 20:35:28.402698 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.402683 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.404649 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.404627 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.404708 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.404670 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.404708 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.404691 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.405267 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405248 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:35:28.405267 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405261 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.405267 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405264 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:35:28.405429 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405287 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:35:28.405962 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405950 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.406015 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405975 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.406015 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.405986 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.407496 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.407478 2577 policy_none.go:50] "Start" Oct 30 20:35:28.407568 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.407503 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:35:28.407568 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.407518 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:35:28.409562 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.409549 2577 policy_none.go:44] "Start" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.445767 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453023 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.453050 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453061 2577 server.go:87] "Starting device plugin registration server" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453309 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453320 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453423 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453498 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.453504 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.454037 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.454097 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.459031 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.460282 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.460301 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.460323 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:35:28.464961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.460354 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:35:28.521064 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.521012 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-115.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:35:28.521064 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.521017 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-115.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:35:28.522218 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.521035 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-115.us-west-1.compute.internal.18e36900b48dd1e9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-115.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-115.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:35:28.373699049 +0000 UTC m=+0.381560561,LastTimestamp:2026-10-30 20:35:28.373699049 +0000 UTC m=+0.381560561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-115.us-west-1.compute.internal,}" Oct 30 20:35:28.553536 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.553458 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.554484 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.554464 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.554575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.554500 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.554575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.554510 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.554575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.554533 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.561065 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.561046 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal"] Oct 30 20:35:28.561118 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.561103 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.561885 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.561866 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.561967 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.561895 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.561967 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.561912 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.564133 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564118 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.564301 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.564358 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564316 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.564871 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564856 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.564938 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564878 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.564938 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564878 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.564938 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564890 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.564938 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564902 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.564938 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.564918 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.566904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.566890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.566969 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.566917 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.567669 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.567651 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.567758 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.567679 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.567758 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.567689 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.585814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.585670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.585903 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.585822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.585903 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.585841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/124b8400391b2921b52ea70b658a260b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"124b8400391b2921b52ea70b658a260b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.594176 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.594153 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.596136 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.596118 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.600718 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.600703 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:35:28.600766 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.600740 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkv5m" Oct 30 20:35:28.623832 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.623744 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-115.us-west-1.compute.internal.18e36900b6661f08 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-115.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-115.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-115.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:35:28.404651784 +0000 UTC m=+0.412513298,LastTimestamp:2026-10-30 20:35:28.404651784 +0000 UTC m=+0.412513298,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-115.us-west-1.compute.internal,}" Oct 30 20:35:28.624470 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.624445 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:35:28.624470 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.624448 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:35:28.624561 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.624448 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 20:35:28.624561 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.624450 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-115.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:35:28.684077 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.684043 2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-115.us-west-1.compute.internal\" not found" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.686284 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.686264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.686391 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.686296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.686391 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.686322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/124b8400391b2921b52ea70b658a260b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"124b8400391b2921b52ea70b658a260b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.686391 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.686378 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/124b8400391b2921b52ea70b658a260b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"124b8400391b2921b52ea70b658a260b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.686391 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.686377 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.686539 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.686377 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.828199 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.828120 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-115.us-west-1.compute.internal\" not found" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.884938 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.884899 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:28.885906 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.885891 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:28.885971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.885922 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:28.885971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.885938 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:28.886031 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.885978 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.898138 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.898118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.901289 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.901261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.947270 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:28.947246 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:28.947270 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:28.947277 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-115.us-west-1.compute.internal\": node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:29.075958 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:29.075926 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.357066 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.357039 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.457342 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.457311 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.496221 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.496181 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:35:30.496745 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.496354 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a41523619245140049339d55e6c91ffd-78a63a0feb3ef4fb.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-115.us-west-1.compute.internal.18e36900b6661f08\": read tcp 10.0.141.115:53296->3.18.77.245:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-115.us-west-1.compute.internal.18e36900b6661f08 default 4786 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-115.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-115.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-115.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:35:28 +0000 UTC,LastTimestamp:2026-10-30 20:35:28.561883187 +0000 UTC m=+0.569744708,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-115.us-west-1.compute.internal,}" Oct 30 20:35:30.557633 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.557601 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.658163 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.658139 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.758987 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.758963 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.786191 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.786153 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:30:29 +0000 UTC" deadline="2028-05-28 11:19:58.18395955 +0000 UTC" Oct 30 20:35:30.786191 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.786184 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13814h44m27.397778826s" Oct 30 20:35:30.823041 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.823017 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:35:30.823965 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.823951 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:35:30.824026 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.823982 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:35:30.824026 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.823994 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:35:30.848013 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.847988 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:35:30.859298 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.859281 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.945895 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.945813 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:35:30.959892 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:30.959865 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:30.994037 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:30.994015 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:35:31.060655 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.060429 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:31.161032 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.160998 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:31.190755 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:31.190725 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124b8400391b2921b52ea70b658a260b.slice/crio-4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590 WatchSource:0}: Error finding container 4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590: Status 404 returned error can't find the container with id 4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590 Oct 30 20:35:31.191125 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:31.191110 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28735f4653fbe0f3b3d64a88696fbee4.slice/crio-1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39 WatchSource:0}: Error finding container 1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39: Status 404 returned error can't find the container with id 1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39 Oct 30 20:35:31.195991 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.195940 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:35:31.261888 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.261853 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:35:31.334920 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.334890 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:35:31.364889 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.364868 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:31.426082 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.426056 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:35:31.427053 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.427035 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:35:31.488788 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.488709 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:35:31.540087 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.540054 2577 apiserver.go:52] "Watching apiserver" Oct 30 20:35:31.564288 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.564253 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:35:31.642257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.642223 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:35:31.642584 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.642562 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-7bt9s","openshift-image-registry/node-ca-7dvbw","openshift-multus/multus-additional-cni-plugins-4fv75","openshift-ovn-kubernetes/ovnkube-node-27nvb","kube-system/global-pull-secret-syncer-2kjtc","kube-system/konnectivity-agent-l2csm","openshift-cluster-node-tuning-operator/tuned-wfcwm","openshift-multus/multus-r7m68","openshift-network-diagnostics/network-check-target-jdq2q","openshift-dns/node-resolver-7kw6v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-8ngvj","kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c"] Oct 30 20:35:31.646292 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.646269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.648363 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.648335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.650500 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.650485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.653892 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.653872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.656099 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.656081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.656184 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.656140 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:31.658184 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.658169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.660672 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.660651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:31.660742 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.660711 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:31.662905 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.662890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:31.662975 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.662958 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:31.665163 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.665146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.667369 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.667353 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.669760 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.669740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.671996 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.671958 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"28735f4653fbe0f3b3d64a88696fbee4","Type":"ContainerStarted","Data":"1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39"} Oct 30 20:35:31.672071 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.672005 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"124b8400391b2921b52ea70b658a260b","Type":"ContainerStarted","Data":"4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590"} Oct 30 20:35:31.672124 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.672077 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.676782 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.676848 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-ovn\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.676848 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-modprobe-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.676915 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkl9\" (UniqueName: \"kubernetes.io/projected/1ea47fc6-4d87-4c4f-b623-801cfe65c047-kube-api-access-djkl9\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.676915 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.676973 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676934 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-k8s-cni-cncf-io\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.676973 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-systemd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677033 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.676974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975f6dec-89de-4c6e-ab23-a665d5d76ced-konnectivity-ca\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.677064 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-sys\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.677107 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-tuned\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.677149 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.677188 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-cni-binary-copy\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.677223 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-kubelet\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677259 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-slash\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677292 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677331 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:31.677366 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-multus-daemon-config\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.677366 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-bin\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677452 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znd9n\" (UniqueName: \"kubernetes.io/projected/b3970765-0603-41f1-9802-a252c08fdce7-kube-api-access-znd9n\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:31.677452 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-kubelet-config\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.677452 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-dbus\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.677539 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-config\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677539 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7092030c-f398-4180-9641-62b31400359a-host\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.677539 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-host\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.677539 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-netns\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.677650 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677650 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-var-lib-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677650 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-netd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677650 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-conf-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.677779 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-systemd-units\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677779 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkvb\" (UniqueName: \"kubernetes.io/projected/7092030c-f398-4180-9641-62b31400359a-kube-api-access-cjkvb\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.677841 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-node-log\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677841 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975f6dec-89de-4c6e-ab23-a665d5d76ced-agent-certs\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.677895 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-system-cni-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.677926 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-script-lib\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.677926 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.677984 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-run\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.677984 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-etc-kubernetes\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678047 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.677994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfj6\" (UniqueName: \"kubernetes.io/projected/f24910a3-693b-4290-aec0-7539fe0115d7-kube-api-access-fsfj6\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678047 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-var-lib-kubelet\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-tmp\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678080 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-kubernetes\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-conf\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678233 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7092030c-f398-4180-9641-62b31400359a-serviceca\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.678233 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-socket-dir-parent\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678233 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rphqg\" (UniqueName: \"kubernetes.io/projected/4fa2e4fe-41cb-4690-bba9-206c3aef048a-kube-api-access-rphqg\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.678233 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-netns\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.678345 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovn-node-metrics-cert\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.678345 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysconfig\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678345 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-hostroot\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678446 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpbk\" (UniqueName: \"kubernetes.io/projected/e933dad7-264e-49fc-ae45-e6ba8ffad936-kube-api-access-fkpbk\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.678446 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.678446 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.678446 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.678575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-cnibin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-log-socket\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.678575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-env-overrides\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.678575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-multus\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678575 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-multus-certs\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-etc-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.678721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-systemd\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-lib-modules\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.678721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cnibin\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.678721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.678930 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:31.678930 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-system-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678930 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-kubelet\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.678930 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.678783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-bin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.698337 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.698316 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.698439 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.698332 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l8lvp\"" Oct 30 20:35:31.699492 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.699472 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.699692 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.699676 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:35:31.699784 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.699755 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.699868 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.699851 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.700104 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.700090 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b4p7g\"" Oct 30 20:35:31.700224 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.700208 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:35:31.702219 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.702206 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:35:31.742719 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.742653 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:35:31.742988 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.742957 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:35:31.742988 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.742963 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:35:31.743197 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.742997 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.743197 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743001 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:35:31.743197 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743004 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mx55q\"" Oct 30 20:35:31.743197 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.742963 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.743197 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743023 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:35:31.743197 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743064 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:35:31.743430 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743221 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8v9jn\"" Oct 30 20:35:31.743430 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743239 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.743430 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743248 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hbdn4\"" Oct 30 20:35:31.743430 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743272 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k4gm7\"" Oct 30 20:35:31.743561 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743454 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdv7w\"" Oct 30 20:35:31.743561 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743486 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.743968 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743951 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:35:31.744013 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.743966 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2mcfh\"" Oct 30 20:35:31.745097 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.745081 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.747557 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747542 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:35:31.747608 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747574 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:35:31.747749 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747730 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:35:31.747749 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747737 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.747914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747783 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.747914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747800 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.747914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747809 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:35:31.747914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747808 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:35:31.747914 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747852 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xq4tq" Oct 30 20:35:31.748054 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.747920 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:35:31.748082 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.748054 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k65fh\"" Oct 30 20:35:31.768269 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.766948 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:35:31.778986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.778962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znd9n\" (UniqueName: \"kubernetes.io/projected/b3970765-0603-41f1-9802-a252c08fdce7-kube-api-access-znd9n\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:31.779088 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.778989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-kubelet-config\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.779088 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-dbus\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.779088 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-config\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.779088 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-registration-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.779088 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af942c5a-661e-423c-b24b-a7a4a1db3bd9-host-slash\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.779088 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7092030c-f398-4180-9641-62b31400359a-host\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779095 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-kubelet-config\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779131 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7092030c-f398-4180-9641-62b31400359a-host\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-host\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-netns\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-host\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-var-lib-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779226 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-netns\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779228 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-dbus\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-var-lib-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779265 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-netd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af942c5a-661e-423c-b24b-a7a4a1db3bd9-iptables-alerter-script\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-conf-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.779362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779354 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-netd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-systemd-units\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkvb\" (UniqueName: \"kubernetes.io/projected/7092030c-f398-4180-9641-62b31400359a-kube-api-access-cjkvb\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-conf-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779444 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-systemd-units\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-node-log\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-node-log\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975f6dec-89de-4c6e-ab23-a665d5d76ced-agent-certs\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-hosts-file\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-system-cni-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-script-lib\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-system-cni-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-sys-fs\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-config\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-run\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-etc-kubernetes\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779787 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:35:31.780028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779824 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779842 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-run\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779857 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-etc-kubernetes\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfj6\" (UniqueName: \"kubernetes.io/projected/f24910a3-693b-4290-aec0-7539fe0115d7-kube-api-access-fsfj6\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-var-lib-kubelet\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-tmp\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-kubernetes\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-conf\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-var-lib-kubelet\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.779991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7092030c-f398-4180-9641-62b31400359a-serviceca\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-kubernetes\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-socket-dir-parent\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphqg\" (UniqueName: \"kubernetes.io/projected/4fa2e4fe-41cb-4690-bba9-206c3aef048a-kube-api-access-rphqg\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780103 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-conf\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780109 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-script-lib\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-netns\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovn-node-metrics-cert\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780163 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-netns\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.780908 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysconfig\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-hostroot\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpbk\" (UniqueName: \"kubernetes.io/projected/e933dad7-264e-49fc-ae45-e6ba8ffad936-kube-api-access-fkpbk\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5qs\" (UniqueName: \"kubernetes.io/projected/af942c5a-661e-423c-b24b-a7a4a1db3bd9-kube-api-access-zk5qs\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780361 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-hostroot\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-cnibin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780396 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7092030c-f398-4180-9641-62b31400359a-serviceca\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-socket-dir-parent\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-cnibin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780478 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysconfig\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-log-socket\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-env-overrides\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.780666 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-tmp-dir\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.781768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-socket-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780706 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-log-socket\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.780743 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret podName:6ed71fea-5f6c-4ba4-be09-b524c1abfc4a nodeName:}" failed. No retries permitted until 2026-10-30 20:35:32.280706782 +0000 UTC m=+3.108365120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret") pod "global-pull-secret-syncer-2kjtc" (UID: "6ed71fea-5f6c-4ba4-be09-b524c1abfc4a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-multus\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-multus-certs\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-etc-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-systemd\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-multus\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-lib-modules\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cnibin\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780915 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-etc-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-system-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-systemd\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.782619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-kubelet\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-bin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781064 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cnibin\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781062 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-env-overrides\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-system-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-ovn\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781138 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-ovn\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781143 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-kubelet\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.780965 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-multus-certs\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781114 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-bin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvwz\" (UniqueName: \"kubernetes.io/projected/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kube-api-access-ztvwz\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-modprobe-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781267 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-lib-modules\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781286 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-modprobe-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.783397 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkl9\" (UniqueName: \"kubernetes.io/projected/1ea47fc6-4d87-4c4f-b623-801cfe65c047-kube-api-access-djkl9\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-k8s-cni-cncf-io\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781450 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-k8s-cni-cncf-io\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-systemd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975f6dec-89de-4c6e-ab23-a665d5d76ced-konnectivity-ca\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28kd\" (UniqueName: \"kubernetes.io/projected/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-kube-api-access-f28kd\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-device-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-sys\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-tuned\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-cni-binary-copy\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-kubelet\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-slash\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781898 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.784234 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-multus-daemon-config\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.781986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-bin\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782077 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975f6dec-89de-4c6e-ab23-a665d5d76ced-konnectivity-ca\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782178 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-slash\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782187 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-sys\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782227 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782279 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-systemd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-bin\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.782344 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782349 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-kubelet\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.782398 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:35:32.282381351 +0000 UTC m=+3.110039680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782584 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-cni-binary-copy\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.782719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-multus-daemon-config\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.783346 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-tmp\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.784710 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.783462 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovn-node-metrics-cert\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.785179 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.783518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975f6dec-89de-4c6e-ab23-a665d5d76ced-agent-certs\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.785179 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.784313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-tuned\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.838567 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.838539 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:35:31.838567 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.838566 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:35:31.838771 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.838579 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5xntv for pod openshift-network-diagnostics/network-check-target-jdq2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:31.838771 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:31.838666 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv podName:5f48b68d-7d92-4417-915f-785bb3e457ab nodeName:}" failed. No retries permitted until 2026-10-30 20:35:32.338645505 +0000 UTC m=+3.166303838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5xntv" (UniqueName: "kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv") pod "network-check-target-jdq2q" (UID: "5f48b68d-7d92-4417-915f-785bb3e457ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:31.840086 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.840059 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkvb\" (UniqueName: \"kubernetes.io/projected/7092030c-f398-4180-9641-62b31400359a-kube-api-access-cjkvb\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.840271 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.840253 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znd9n\" (UniqueName: \"kubernetes.io/projected/b3970765-0603-41f1-9802-a252c08fdce7-kube-api-access-znd9n\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:31.840465 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.840443 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphqg\" (UniqueName: \"kubernetes.io/projected/4fa2e4fe-41cb-4690-bba9-206c3aef048a-kube-api-access-rphqg\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.840619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.840603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpbk\" (UniqueName: \"kubernetes.io/projected/e933dad7-264e-49fc-ae45-e6ba8ffad936-kube-api-access-fkpbk\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:31.840677 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.840630 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfj6\" (UniqueName: \"kubernetes.io/projected/f24910a3-693b-4290-aec0-7539fe0115d7-kube-api-access-fsfj6\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.841239 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.841223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkl9\" (UniqueName: \"kubernetes.io/projected/1ea47fc6-4d87-4c4f-b623-801cfe65c047-kube-api-access-djkl9\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.882421 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882421 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvwz\" (UniqueName: \"kubernetes.io/projected/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kube-api-access-ztvwz\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28kd\" (UniqueName: \"kubernetes.io/projected/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-kube-api-access-f28kd\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-device-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-registration-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af942c5a-661e-423c-b24b-a7a4a1db3bd9-host-slash\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af942c5a-661e-423c-b24b-a7a4a1db3bd9-iptables-alerter-script\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-device-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882566 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-hosts-file\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.882599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af942c5a-661e-423c-b24b-a7a4a1db3bd9-host-slash\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882576 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-registration-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-sys-fs\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882625 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-hosts-file\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5qs\" (UniqueName: \"kubernetes.io/projected/af942c5a-661e-423c-b24b-a7a4a1db3bd9-kube-api-access-zk5qs\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-sys-fs\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-tmp-dir\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-socket-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.882995 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.882811 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-socket-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.883738 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.883718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-tmp-dir\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.883855 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.883839 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af942c5a-661e-423c-b24b-a7a4a1db3bd9-iptables-alerter-script\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.943237 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.943208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5qs\" (UniqueName: \"kubernetes.io/projected/af942c5a-661e-423c-b24b-a7a4a1db3bd9-kube-api-access-zk5qs\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:31.943343 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.943285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28kd\" (UniqueName: \"kubernetes.io/projected/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-kube-api-access-f28kd\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.943343 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.943306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvwz\" (UniqueName: \"kubernetes.io/projected/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kube-api-access-ztvwz\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:31.957889 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.957871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:35:31.964140 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.964127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7m68" Oct 30 20:35:31.967828 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:31.967804 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea47fc6_4d87_4c4f_b623_801cfe65c047.slice/crio-f79192217c95cbb57162e1f50fe40e18f61965d6f2826ae0774723bf39c1e860 WatchSource:0}: Error finding container f79192217c95cbb57162e1f50fe40e18f61965d6f2826ae0774723bf39c1e860: Status 404 returned error can't find the container with id f79192217c95cbb57162e1f50fe40e18f61965d6f2826ae0774723bf39c1e860 Oct 30 20:35:31.972141 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:31.972120 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24910a3_693b_4290_aec0_7539fe0115d7.slice/crio-dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc WatchSource:0}: Error finding container dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc: Status 404 returned error can't find the container with id dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc Oct 30 20:35:31.987289 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.987268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:31.989387 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.989367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:35:31.991120 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.991103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:35:31.997493 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:31.997472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:35:31.998536 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:31.998514 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975f6dec_89de_4c6e_ab23_a665d5d76ced.slice/crio-aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5 WatchSource:0}: Error finding container aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5: Status 404 returned error can't find the container with id aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5 Oct 30 20:35:32.001318 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:32.001237 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa2e4fe_41cb_4690_bba9_206c3aef048a.slice/crio-66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d WatchSource:0}: Error finding container 66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d: Status 404 returned error can't find the container with id 66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d Oct 30 20:35:32.004897 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:32.004877 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7092030c_f398_4180_9641_62b31400359a.slice/crio-01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56 WatchSource:0}: Error finding container 01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56: Status 404 returned error can't find the container with id 01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56 Oct 30 20:35:32.006498 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:32.006466 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b875558_4e2e_4eaf_9dc6_7861f1a95be0.slice/crio-740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a WatchSource:0}: Error finding container 740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a: Status 404 returned error can't find the container with id 740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a Oct 30 20:35:32.007069 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.007053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:35:32.009669 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.009651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:35:32.011899 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.011881 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:32.017588 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:32.017563 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf942c5a_661e_423c_b24b_a7a4a1db3bd9.slice/crio-939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a WatchSource:0}: Error finding container 939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a: Status 404 returned error can't find the container with id 939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a Oct 30 20:35:32.020905 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:32.020840 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7257f1_1a1e_4029_8c65_f9c8e5ec72f2.slice/crio-5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4 WatchSource:0}: Error finding container 5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4: Status 404 returned error can't find the container with id 5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4 Oct 30 20:35:32.023821 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:35:32.023800 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode933dad7_264e_49fc_ae45_e6ba8ffad936.slice/crio-41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b WatchSource:0}: Error finding container 41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b: Status 404 returned error can't find the container with id 41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b Oct 30 20:35:32.286515 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.286243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:32.286515 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.286313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:32.286515 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.286368 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:32.286515 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.286422 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:32.286515 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.286450 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret podName:6ed71fea-5f6c-4ba4-be09-b524c1abfc4a nodeName:}" failed. No retries permitted until 2026-10-30 20:35:33.286429613 +0000 UTC m=+4.114087943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret") pod "global-pull-secret-syncer-2kjtc" (UID: "6ed71fea-5f6c-4ba4-be09-b524c1abfc4a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:32.286515 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.286471 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:35:33.286461602 +0000 UTC m=+4.114119923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:32.387312 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.387277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:32.387552 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.387472 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:35:32.387552 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.387490 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:35:32.387552 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.387510 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5xntv for pod openshift-network-diagnostics/network-check-target-jdq2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:32.387710 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:32.387568 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv podName:5f48b68d-7d92-4417-915f-785bb3e457ab nodeName:}" failed. No retries permitted until 2026-10-30 20:35:33.387551912 +0000 UTC m=+4.215210227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xntv" (UniqueName: "kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv") pod "network-check-target-jdq2q" (UID: "5f48b68d-7d92-4417-915f-785bb3e457ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:32.647955 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.647917 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" event={"ID":"1ea47fc6-4d87-4c4f-b623-801cfe65c047","Type":"ContainerStarted","Data":"f79192217c95cbb57162e1f50fe40e18f61965d6f2826ae0774723bf39c1e860"} Oct 30 20:35:32.649156 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.649130 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b"} Oct 30 20:35:32.650321 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.650299 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ngvj" event={"ID":"af942c5a-661e-423c-b24b-a7a4a1db3bd9","Type":"ContainerStarted","Data":"939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a"} Oct 30 20:35:32.651602 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.651568 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7dvbw" event={"ID":"7092030c-f398-4180-9641-62b31400359a","Type":"ContainerStarted","Data":"01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56"} Oct 30 20:35:32.652797 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.652771 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerStarted","Data":"66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d"} Oct 30 20:35:32.653969 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.653949 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7m68" event={"ID":"f24910a3-693b-4290-aec0-7539fe0115d7","Type":"ContainerStarted","Data":"dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc"} Oct 30 20:35:32.655157 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.655130 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" event={"ID":"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2","Type":"ContainerStarted","Data":"5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4"} Oct 30 20:35:32.656538 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.656515 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7kw6v" event={"ID":"5b875558-4e2e-4eaf-9dc6-7861f1a95be0","Type":"ContainerStarted","Data":"740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a"} Oct 30 20:35:32.657702 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.657682 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2csm" event={"ID":"975f6dec-89de-4c6e-ab23-a665d5d76ced","Type":"ContainerStarted","Data":"aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5"} Oct 30 20:35:32.750500 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.750461 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:30:31 +0000 UTC" deadline="2028-04-11 12:38:56.10395318 +0000 UTC" Oct 30 20:35:32.750500 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:32.750500 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12688h3m23.353458225s" Oct 30 20:35:33.293699 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.293285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:33.293699 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.293358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:33.293699 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.293508 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:33.293699 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.293567 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:35:35.293549233 +0000 UTC m=+6.121207565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:33.293699 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.293636 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:33.293699 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.293669 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret podName:6ed71fea-5f6c-4ba4-be09-b524c1abfc4a nodeName:}" failed. No retries permitted until 2026-10-30 20:35:35.293659378 +0000 UTC m=+6.121317699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret") pod "global-pull-secret-syncer-2kjtc" (UID: "6ed71fea-5f6c-4ba4-be09-b524c1abfc4a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:33.395168 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.394796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:33.395168 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.395028 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:35:33.395168 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.395049 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:35:33.395168 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.395071 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5xntv for pod openshift-network-diagnostics/network-check-target-jdq2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:33.395168 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.395130 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv podName:5f48b68d-7d92-4417-915f-785bb3e457ab nodeName:}" failed. No retries permitted until 2026-10-30 20:35:35.395113771 +0000 UTC m=+6.222772100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xntv" (UniqueName: "kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv") pod "network-check-target-jdq2q" (UID: "5f48b68d-7d92-4417-915f-785bb3e457ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:33.642902 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.642532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:33.642902 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.642659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:33.642902 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.642749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:33.642902 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.642832 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:33.643499 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.643299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:33.643499 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:33.643397 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:33.751727 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.751636 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:30:31 +0000 UTC" deadline="2028-07-21 16:26:13.91402477 +0000 UTC" Oct 30 20:35:33.751727 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:33.751694 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15115h50m40.162334054s" Oct 30 20:35:35.312839 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.312133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:35.312839 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.312459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:35.312839 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.312606 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:35.312839 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.312682 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:35:39.312663628 +0000 UTC m=+10.140321962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:35.312839 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.312763 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:35.312839 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.312800 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret podName:6ed71fea-5f6c-4ba4-be09-b524c1abfc4a nodeName:}" failed. No retries permitted until 2026-10-30 20:35:39.31278868 +0000 UTC m=+10.140447000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret") pod "global-pull-secret-syncer-2kjtc" (UID: "6ed71fea-5f6c-4ba4-be09-b524c1abfc4a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:35.413492 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.413152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:35.413492 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.413341 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:35:35.413492 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.413364 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:35:35.413492 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.413377 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5xntv for pod openshift-network-diagnostics/network-check-target-jdq2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:35.413492 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.413455 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv podName:5f48b68d-7d92-4417-915f-785bb3e457ab nodeName:}" failed. No retries permitted until 2026-10-30 20:35:39.413436348 +0000 UTC m=+10.241094687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xntv" (UniqueName: "kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv") pod "network-check-target-jdq2q" (UID: "5f48b68d-7d92-4417-915f-785bb3e457ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:35.643641 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.643605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:35.643815 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.643704 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:35.643815 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.643746 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:35.643928 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.643816 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:35.643928 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.643859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:35.644052 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:35.643927 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:35.707113 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.707071 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="6960e9e81a926f3f80627850921f673fe15477ca8b93f9fe56d61a60526a65e4" exitCode=0 Oct 30 20:35:35.707261 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.707183 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerDied","Data":"6960e9e81a926f3f80627850921f673fe15477ca8b93f9fe56d61a60526a65e4"} Oct 30 20:35:35.712230 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.712201 2577 generic.go:356] "Generic (PLEG): container finished" podID="28735f4653fbe0f3b3d64a88696fbee4" containerID="2a1463e9b6c511d9e225c2035bf7c47a7d86627c94ff5c378053ac7952694382" exitCode=0 Oct 30 20:35:35.712322 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.712285 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"28735f4653fbe0f3b3d64a88696fbee4","Type":"ContainerDied","Data":"2a1463e9b6c511d9e225c2035bf7c47a7d86627c94ff5c378053ac7952694382"} Oct 30 20:35:35.721318 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:35.721289 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"124b8400391b2921b52ea70b658a260b","Type":"ContainerStarted","Data":"a1ccb0ddf6524571724be344192204fc913e630f44075c6e7a4fc60f7194d215"} Oct 30 20:35:36.740649 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:36.740522 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"28735f4653fbe0f3b3d64a88696fbee4","Type":"ContainerStarted","Data":"8d333249fb913757e581e3efd2a96c167a273b593b68b7302d9b4f33d80ebf18"} Oct 30 20:35:36.856066 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:36.855896 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" podStartSLOduration=5.855875148 podStartE2EDuration="5.855875148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:36.854991349 +0000 UTC m=+7.682649688" watchObservedRunningTime="2026-10-30 20:35:36.855875148 +0000 UTC m=+7.683533487" Oct 30 20:35:36.856219 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:36.856070 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" podStartSLOduration=5.856060764 podStartE2EDuration="5.856060764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:36.060387029 +0000 UTC m=+6.888045373" watchObservedRunningTime="2026-10-30 20:35:36.856060764 +0000 UTC m=+7.683719111" Oct 30 20:35:37.643711 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:37.643679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:37.643886 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:37.643799 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:37.644165 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:37.644122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:37.644264 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:37.644211 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:37.644264 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:37.644252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:37.644363 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:37.644293 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:37.748089 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:37.748045 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2csm" event={"ID":"975f6dec-89de-4c6e-ab23-a665d5d76ced","Type":"ContainerStarted","Data":"161f1d55928b3eb51188e5f41404e27c26e9c0f16a5a16cc297f00e76a0a0f7a"} Oct 30 20:35:37.863556 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:37.863470 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l2csm" podStartSLOduration=3.022474755 podStartE2EDuration="7.863451551s" totalImagesPullingTime="4.840976796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:37.862742968 +0000 UTC m=+8.690401306" watchObservedRunningTime="2026-10-30 20:35:37.863451551 +0000 UTC m=+8.691109890" Oct 30 20:35:38.752553 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:38.752512 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7dvbw" event={"ID":"7092030c-f398-4180-9641-62b31400359a","Type":"ContainerStarted","Data":"59c66c57be913d97f6d50bd4b36a7dc740baa23b06703fa6b58147c44600df45"} Oct 30 20:35:38.757439 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:38.757348 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" event={"ID":"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2","Type":"ContainerStarted","Data":"426085297c1f76e46b6846b1c34bbf850286e21e3edb075c9a109ca9daec9891"} Oct 30 20:35:39.349479 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:39.349443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:39.349657 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:39.349519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:39.349657 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.349626 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:39.349739 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.349673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:35:47.349660362 +0000 UTC m=+18.177318677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:39.350144 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.349958 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:39.350144 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.350008 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret podName:6ed71fea-5f6c-4ba4-be09-b524c1abfc4a nodeName:}" failed. No retries permitted until 2026-10-30 20:35:47.349994019 +0000 UTC m=+18.177652337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret") pod "global-pull-secret-syncer-2kjtc" (UID: "6ed71fea-5f6c-4ba4-be09-b524c1abfc4a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:39.449902 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:39.449861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:39.450074 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.450056 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:35:39.450142 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.450083 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:35:39.450142 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.450096 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5xntv for pod openshift-network-diagnostics/network-check-target-jdq2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:39.450251 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.450155 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv podName:5f48b68d-7d92-4417-915f-785bb3e457ab nodeName:}" failed. No retries permitted until 2026-10-30 20:35:47.450137773 +0000 UTC m=+18.277796090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xntv" (UniqueName: "kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv") pod "network-check-target-jdq2q" (UID: "5f48b68d-7d92-4417-915f-785bb3e457ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:39.645178 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:39.645142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:39.645334 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.645267 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:39.646803 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:39.645666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:39.646803 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.645778 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:39.646803 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:39.645845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:39.646803 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:39.645921 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:40.746650 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:40.745634 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:35:40.762378 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:40.762336 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7m68" event={"ID":"f24910a3-693b-4290-aec0-7539fe0115d7","Type":"ContainerStarted","Data":"3809e39be655f6970fd7b2137a89fc300546043938e873d0d602a04754a7344c"} Oct 30 20:35:40.764675 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:40.764650 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" event={"ID":"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2","Type":"ContainerStarted","Data":"d16f24ceff04003dbf5942e9089f95cac7901a47d39af64e60a1ccf02380146f"} Oct 30 20:35:40.766033 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:40.766009 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7kw6v" event={"ID":"5b875558-4e2e-4eaf-9dc6-7861f1a95be0","Type":"ContainerStarted","Data":"e2751d672261a2f4047b2b20be17dc8373880086a687af8ab5334c7f65d0b2d9"} Oct 30 20:35:40.881165 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:40.880667 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7dvbw" podStartSLOduration=5.094955239 podStartE2EDuration="10.880649449s" totalImagesPullingTime="5.78569421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:38.867828565 +0000 UTC m=+9.695486903" watchObservedRunningTime="2026-10-30 20:35:40.880649449 +0000 UTC m=+11.708307787" Oct 30 20:35:40.996204 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:40.996155 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r7m68" podStartSLOduration=3.218579042 podStartE2EDuration="10.996141459s" totalImagesPullingTime="7.777562417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:40.881995695 +0000 UTC m=+11.709654027" watchObservedRunningTime="2026-10-30 20:35:40.996141459 +0000 UTC m=+11.823799795" Oct 30 20:35:41.014957 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.014927 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:41.638694 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.638579 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:35:40.745649871Z","UUID":"35fc5a68-3760-4dec-b92f-b1c17ada224f","Handler":null,"Name":"","Endpoint":""} Oct 30 20:35:41.640337 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.640306 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:35:41.640337 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.640338 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:35:41.640734 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.640704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:41.640835 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.640711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:41.640891 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:41.640825 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:41.640891 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.640860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:41.641009 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:41.640923 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:41.641331 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:41.641308 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:41.771821 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.771785 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="0e5a01bd4629c7bf118782d8314396d2f3680a1a7d323b4bdae9026a7ea66815" exitCode=0 Oct 30 20:35:41.772313 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.772066 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerDied","Data":"0e5a01bd4629c7bf118782d8314396d2f3680a1a7d323b4bdae9026a7ea66815"} Oct 30 20:35:41.898141 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:41.897682 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7kw6v" podStartSLOduration=3.438217134 podStartE2EDuration="11.897663323s" totalImagesPullingTime="8.459446189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:40.996627089 +0000 UTC m=+11.824285429" watchObservedRunningTime="2026-10-30 20:35:41.897663323 +0000 UTC m=+12.725321663" Oct 30 20:35:42.498321 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:42.498284 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:42.499139 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:42.499113 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:42.775266 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:42.775190 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8ngvj" event={"ID":"af942c5a-661e-423c-b24b-a7a4a1db3bd9","Type":"ContainerStarted","Data":"89864ed682123cc7f5eec2cbfdfe538d225eaa5655a5bf2961ec0399280a3ef2"} Oct 30 20:35:42.776072 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:42.776051 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:35:42.891332 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:42.891272 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8ngvj" podStartSLOduration=3.960151342 podStartE2EDuration="12.891255272s" totalImagesPullingTime="8.93110393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:42.891049003 +0000 UTC m=+13.718707342" watchObservedRunningTime="2026-10-30 20:35:42.891255272 +0000 UTC m=+13.718913609" Oct 30 20:35:43.641025 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.640987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:43.641193 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.640992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:43.641193 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:43.641129 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:43.641258 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:43.641204 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:43.641258 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.641001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:43.641334 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:43.641265 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:43.780223 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.780138 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="6dff6f32b0d90f2aa7b50f9458cf1c84eacdecadf022326bfb2b331da4ac83a1" exitCode=0 Oct 30 20:35:43.780888 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.780226 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerDied","Data":"6dff6f32b0d90f2aa7b50f9458cf1c84eacdecadf022326bfb2b331da4ac83a1"} Oct 30 20:35:43.783267 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.783246 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" event={"ID":"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2","Type":"ContainerStarted","Data":"7a664283ad51039fa166d383777a8f6c9d3a2b308c82764e39e842a72b0670ad"} Oct 30 20:35:43.784779 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:43.784758 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" event={"ID":"1ea47fc6-4d87-4c4f-b623-801cfe65c047","Type":"ContainerStarted","Data":"d7c41e000c1c98b6cdcd0a3543e77b25b0752e20d0aaab0a25bb6d20fe9d9534"} Oct 30 20:35:44.020194 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:44.020139 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" podStartSLOduration=3.397818746 podStartE2EDuration="14.02012381s" totalImagesPullingTime="10.622305064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:44.019722494 +0000 UTC m=+14.847380832" watchObservedRunningTime="2026-10-30 20:35:44.02012381 +0000 UTC m=+14.847782146" Oct 30 20:35:44.182241 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:44.182186 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" podStartSLOduration=3.143601113 podStartE2EDuration="14.182172075s" totalImagesPullingTime="11.038570962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:44.18201698 +0000 UTC m=+15.009675317" watchObservedRunningTime="2026-10-30 20:35:44.182172075 +0000 UTC m=+15.009830412" Oct 30 20:35:45.644353 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:45.644125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:45.644858 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:45.644157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:45.644858 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:45.644157 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:45.644858 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:45.644493 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:45.644858 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:45.644629 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:45.644858 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:45.644696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:45.791974 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:45.791936 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="8d1c778a797b447374e36c5b70273ac0b0c2e4ecc1a0ac5b8c755f19ccfb48fe" exitCode=0 Oct 30 20:35:45.792145 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:45.791992 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerDied","Data":"8d1c778a797b447374e36c5b70273ac0b0c2e4ecc1a0ac5b8c755f19ccfb48fe"} Oct 30 20:35:47.415513 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.415315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:47.415961 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.415563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:47.415961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.415487 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:47.415961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.415640 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.415625816 +0000 UTC m=+34.243284135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:35:47.415961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.415698 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:47.415961 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.415756 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret podName:6ed71fea-5f6c-4ba4-be09-b524c1abfc4a nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.415741192 +0000 UTC m=+34.243399523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret") pod "global-pull-secret-syncer-2kjtc" (UID: "6ed71fea-5f6c-4ba4-be09-b524c1abfc4a") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:35:47.516551 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.516515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:47.516724 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.516703 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:35:47.516780 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.516730 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:35:47.516780 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.516746 2577 projected.go:196] Error preparing data for projected volume kube-api-access-5xntv for pod openshift-network-diagnostics/network-check-target-jdq2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:47.516884 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.516810 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv podName:5f48b68d-7d92-4417-915f-785bb3e457ab nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.516791847 +0000 UTC m=+34.344450167 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xntv" (UniqueName: "kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv") pod "network-check-target-jdq2q" (UID: "5f48b68d-7d92-4417-915f-785bb3e457ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:35:47.641755 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.641724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:47.641886 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.641734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:47.641886 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.641871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:47.641886 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.641864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:47.641993 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.641940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:47.642038 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:47.642021 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:47.800108 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.800029 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"69a9958c8064727e5903834f5a492f14386fe5a5a9e4c708bf3d4dd43aa45feb"} Oct 30 20:35:47.800108 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.800104 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"0b59bc0582ee69d960899595772321a7955259a3ecfc92109a1d3a4b383e01be"} Oct 30 20:35:47.800272 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.800143 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"e0aa0ad699bc6fd3ee7bacdb82c13df287e312dd0e89e62c1af20b4bf2ef8ca7"} Oct 30 20:35:47.800272 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.800172 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"83b6731fb7d3356fb5626a975835c3d285e33f179700af793092be071a44a2e4"} Oct 30 20:35:47.800272 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.800197 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"36e9c0c50629f53a7e80344c228173a9c641b69c4d48f9fe50549c560128dcd1"} Oct 30 20:35:47.800272 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:47.800231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"fe87ab2b7c1ca1b59bbd70942f8971416bad5f50c810069febb985014fa5fb35"} Oct 30 20:35:48.804883 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:48.804846 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="b1e6e74fdd6df69ee02a8d27e860a9bc88f5c58aebccf5c6c61eca5410eb1b57" exitCode=0 Oct 30 20:35:48.805491 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:48.804894 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerDied","Data":"b1e6e74fdd6df69ee02a8d27e860a9bc88f5c58aebccf5c6c61eca5410eb1b57"} Oct 30 20:35:49.641933 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:49.641901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:49.642082 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:49.641984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:49.642082 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:49.642032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:49.642082 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:49.642067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:49.642233 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:49.642119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:49.642233 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:49.642163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:49.809559 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:49.809526 2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="fed5b5e052d9bc06540065aa2143c7df574d1f3921bd81bd6050c0468ef111f4" exitCode=0 Oct 30 20:35:49.810032 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:49.809579 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerDied","Data":"fed5b5e052d9bc06540065aa2143c7df574d1f3921bd81bd6050c0468ef111f4"} Oct 30 20:35:49.812761 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:49.812740 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"57af41c81c5ab6aab6f45df33e45f21904088ae70316dfa4b4b1ae1a86629503"} Oct 30 20:35:50.817640 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:50.817604 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4fv75" event={"ID":"4fa2e4fe-41cb-4690-bba9-206c3aef048a","Type":"ContainerStarted","Data":"8b89c8c590eeb8e9bd1f00a69842df8238800b706851b37d20a04c63f71521f9"} Oct 30 20:35:50.991404 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:50.991351 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4fv75" podStartSLOduration=7.564775879 podStartE2EDuration="20.991339004s" totalImagesPullingTime="13.426563125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:50.991202885 +0000 UTC m=+21.818861222" watchObservedRunningTime="2026-10-30 20:35:50.991339004 +0000 UTC m=+21.818997359" Oct 30 20:35:51.641517 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:51.641349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:51.641655 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:51.641374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:51.641655 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:51.641608 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:51.641749 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:51.641697 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:51.641749 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:51.641443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:51.641869 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:51.641848 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:51.823734 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:51.823695 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" event={"ID":"e933dad7-264e-49fc-ae45-e6ba8ffad936","Type":"ContainerStarted","Data":"d5e4bdebbfa5551a696417ea80ef8a71364c0d98982dc4a1a58ec5388c4718e7"} Oct 30 20:35:52.025557 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:52.025461 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" podStartSLOduration=7.21332774 podStartE2EDuration="22.02544574s" totalImagesPullingTime="14.812118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:51.999365753 +0000 UTC m=+22.827024092" watchObservedRunningTime="2026-10-30 20:35:52.02544574 +0000 UTC m=+22.853104078" Oct 30 20:35:52.837580 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:52.837553 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:52.838153 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:52.838131 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:52.838251 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:52.838185 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:52.853128 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:52.853106 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:52.853283 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:52.853203 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:35:53.641190 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.641162 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:53.641683 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.641222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:53.641683 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.641226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:53.641683 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:53.641313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:53.641683 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:53.641405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:53.641683 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:53.641484 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:53.958293 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.958220 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kjtc"] Oct 30 20:35:53.958760 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.958315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:53.958760 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:53.958388 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:53.960987 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.960961 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jdq2q"] Oct 30 20:35:53.961105 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.961066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:53.961157 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:53.961142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:53.967395 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.967372 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bt9s"] Oct 30 20:35:53.967552 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:53.967489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:53.967639 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:53.967621 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:55.640792 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:55.640764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:55.640792 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:55.640783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:55.641458 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:55.640783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:55.641458 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:55.640868 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:55.641458 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:55.640959 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:55.641458 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:55.641053 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:57.641274 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:57.641247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:57.641765 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:57.641275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:57.641765 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:57.641349 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:57.641765 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:57.641382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:57.641765 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:57.641485 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:57.641765 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:57.641558 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:59.641864 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:59.641689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:35:59.641864 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:59.641766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:35:59.641864 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:59.641807 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jdq2q" podUID="5f48b68d-7d92-4417-915f-785bb3e457ab" Oct 30 20:35:59.641864 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:59.641826 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2kjtc" podUID="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" Oct 30 20:35:59.641864 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:59.641860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:35:59.642489 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:35:59.641921 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bt9s" podUID="b3970765-0603-41f1-9802-a252c08fdce7" Oct 30 20:35:59.940428 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:59.940237 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeReady" Oct 30 20:35:59.940592 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:35:59.940513 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 20:36:00.054867 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.054832 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tkhfn"] Oct 30 20:36:00.080339 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.080310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:00.103738 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.103698 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldcc4"] Oct 30 20:36:00.134591 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.134556 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sg6jp"] Oct 30 20:36:00.134744 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.134676 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:00.150558 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.150537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:00.150558 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.150541 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tmxhb"] Oct 30 20:36:00.153334 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.153233 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.153334 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.153303 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.155610 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.155589 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp"] Oct 30 20:36:00.155731 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.155709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.158305 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.158286 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-789f8dddc5-m74zk"] Oct 30 20:36:00.158429 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.158398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.160521 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.160501 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2"] Oct 30 20:36:00.160661 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.160639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.162199 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.162182 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv"] Oct 30 20:36:00.162327 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.162308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.163984 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.163967 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2lsbc"] Oct 30 20:36:00.164076 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.164064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.167473 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.167456 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz"] Oct 30 20:36:00.167608 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.167590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.171315 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.171294 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zbpwc"] Oct 30 20:36:00.172728 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.172690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:00.184846 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.184825 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-587bd9b797-w6c54"] Oct 30 20:36:00.184966 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.184948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:00.187705 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.187685 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85df8d78bd-v8dx4"] Oct 30 20:36:00.187831 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.187815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.199216 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.199195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5f7r\" (UniqueName: \"kubernetes.io/projected/cacd9cad-031b-4a85-a86d-a04eecb18d95-kube-api-access-z5f7r\") pod \"kas-connection-checker-685895c7c9-tkhfn\" (UID: \"cacd9cad-031b-4a85-a86d-a04eecb18d95\") " pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:00.199216 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.199222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rd8l\" (UniqueName: \"kubernetes.io/projected/70d94d6f-863f-4819-b36f-5d2c753471bd-kube-api-access-9rd8l\") pod \"kas-connection-checker-685895c7c9-ldcc4\" (UID: \"70d94d6f-863f-4819-b36f-5d2c753471bd\") " pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:00.203323 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.203299 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf"] Oct 30 20:36:00.204276 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.204253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.222528 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.222508 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n"] Oct 30 20:36:00.222644 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.222630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.235470 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.235447 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.235578 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.235501 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.236734 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.236712 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 20:36:00.236841 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.236829 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.236875 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.236861 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.237020 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.237009 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.237077 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.237060 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 20:36:00.242877 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.242858 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 20:36:00.243561 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.243545 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4kmkw\"" Oct 30 20:36:00.244345 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.244331 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 20:36:00.244567 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.244552 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8jh5r"] Oct 30 20:36:00.244658 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.244646 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 20:36:00.244700 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.244664 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:00.244858 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.244838 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.245489 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.245471 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 20:36:00.248974 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.248955 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 20:36:00.249170 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.249154 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-57fvr\"" Oct 30 20:36:00.253081 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.253059 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 20:36:00.253178 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.253130 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.253445 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.253426 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 20:36:00.253551 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.253538 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 20:36:00.253641 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.253622 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6lzrx\"" Oct 30 20:36:00.268761 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.268735 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s6pl6"] Oct 30 20:36:00.268988 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.268967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.293094 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293070 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sg6jp"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293105 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tkhfn"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293121 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldcc4"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293134 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293148 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293164 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293182 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293196 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tmxhb"] Oct 30 20:36:00.293248 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.293196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.298598 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.298570 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.298598 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.298589 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.298774 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.298651 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7m9rl\"" Oct 30 20:36:00.299190 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299169 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 20:36:00.299670 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55m2\" (UniqueName: \"kubernetes.io/projected/cd681674-072e-4dff-a369-9d956c32b960-kube-api-access-l55m2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.299755 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299684 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.299755 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-snapshots\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.299834 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.299867 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.299867 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-serving-cert\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.299925 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c365d189-b04a-4b12-8519-30c074796180-serving-cert\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.299925 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.299994 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.299955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd681674-072e-4dff-a369-9d956c32b960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.300030 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.300065 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqdc\" (UniqueName: \"kubernetes.io/projected/c365d189-b04a-4b12-8519-30c074796180-kube-api-access-frqdc\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.300112 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.300146 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d68f309-a728-4000-a631-86ef2c50b3e0-serving-cert\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.300198 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.300232 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqr4\" (UniqueName: \"kubernetes.io/projected/3d68f309-a728-4000-a631-86ef2c50b3e0-kube-api-access-7hqr4\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.300280 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:00.300312 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.300348 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.300390 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.300454 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktzm\" (UniqueName: \"kubernetes.io/projected/470aba85-18f4-4acc-9e98-6233c2ae3c9c-kube-api-access-4ktzm\") pod \"kas-connection-checker-685895c7c9-sg6jp\" (UID: \"470aba85-18f4-4acc-9e98-6233c2ae3c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:00.300496 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-service-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.300536 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd681674-072e-4dff-a369-9d956c32b960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.300570 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-tmp\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.300606 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.300647 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.300681 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-default-certificate\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.300728 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcp7p\" (UniqueName: \"kubernetes.io/projected/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-kube-api-access-fcp7p\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.300811 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300799 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-config\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.300850 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-config\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.300884 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea62097-316d-49f5-bccf-dba48029bb6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.300884 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.300964 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.300964 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qrj\" (UniqueName: \"kubernetes.io/projected/f340b19f-a26a-4514-8785-f0714fc17696-kube-api-access-j5qrj\") pod \"volume-data-source-validator-58c9f5f46d-7sjnz\" (UID: \"f340b19f-a26a-4514-8785-f0714fc17696\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:00.301041 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.300967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f7r\" (UniqueName: \"kubernetes.io/projected/cacd9cad-031b-4a85-a86d-a04eecb18d95-kube-api-access-z5f7r\") pod \"kas-connection-checker-685895c7c9-tkhfn\" (UID: \"cacd9cad-031b-4a85-a86d-a04eecb18d95\") " pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:00.301041 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.301111 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.301152 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.301152 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdngs\" (UniqueName: \"kubernetes.io/projected/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-kube-api-access-sdngs\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.301249 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6xs\" (UniqueName: \"kubernetes.io/projected/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-kube-api-access-bv6xs\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.301249 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rd8l\" (UniqueName: \"kubernetes.io/projected/70d94d6f-863f-4819-b36f-5d2c753471bd-kube-api-access-9rd8l\") pod \"kas-connection-checker-685895c7c9-ldcc4\" (UID: \"70d94d6f-863f-4819-b36f-5d2c753471bd\") " pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:00.301385 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhrc\" (UniqueName: \"kubernetes.io/projected/7ea62097-316d-49f5-bccf-dba48029bb6d-kube-api-access-pjhrc\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.301480 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.301536 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.301588 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-stats-auth\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.301630 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db8922f9-31fe-4afd-b227-313eecf0fcaa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:00.301668 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.301668 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.301768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.301768 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.301750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.302319 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.302305 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf"] Oct 30 20:36:00.302965 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.302950 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zbpwc"] Oct 30 20:36:00.325574 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.325546 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 20:36:00.325951 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.325935 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 20:36:00.326285 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326266 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:36:00.326323 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326315 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 20:36:00.326369 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326315 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rgvdg\"" Oct 30 20:36:00.326369 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326329 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-79tcj\"" Oct 30 20:36:00.326441 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326371 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 20:36:00.326441 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326390 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pwzv\"" Oct 30 20:36:00.326546 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326535 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qrh4r\"" Oct 30 20:36:00.326604 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.326590 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cn78n\"" Oct 30 20:36:00.327486 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.327474 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hxg8n\"" Oct 30 20:36:00.327692 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.327679 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rtr46\"" Oct 30 20:36:00.327726 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.327713 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 20:36:00.327878 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.327867 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 20:36:00.332396 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.332382 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 20:36:00.333649 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333632 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.333754 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333738 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 20:36:00.333812 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333740 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.333812 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333780 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.333900 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333842 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.333967 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333949 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.334027 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.333989 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 20:36:00.334079 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.334032 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.334079 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.334034 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 20:36:00.335488 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.335473 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:36:00.335584 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.335492 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.335865 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.335849 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.337823 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.337810 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.338793 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.338778 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vdhsx\"" Oct 30 20:36:00.343539 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.343510 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 20:36:00.344163 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.344149 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 20:36:00.377707 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.377639 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 20:36:00.381099 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.381067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rd8l\" (UniqueName: \"kubernetes.io/projected/70d94d6f-863f-4819-b36f-5d2c753471bd-kube-api-access-9rd8l\") pod \"kas-connection-checker-685895c7c9-ldcc4\" (UID: \"70d94d6f-863f-4819-b36f-5d2c753471bd\") " pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:00.381250 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.381067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f7r\" (UniqueName: \"kubernetes.io/projected/cacd9cad-031b-4a85-a86d-a04eecb18d95-kube-api-access-z5f7r\") pod \"kas-connection-checker-685895c7c9-tkhfn\" (UID: \"cacd9cad-031b-4a85-a86d-a04eecb18d95\") " pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:00.382916 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.382890 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2lsbc"] Oct 30 20:36:00.387548 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.387507 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jh5r"] Oct 30 20:36:00.389939 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.389912 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-789f8dddc5-m74zk"] Oct 30 20:36:00.390064 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.389968 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6pl6"] Oct 30 20:36:00.396596 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.396558 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-587bd9b797-w6c54"] Oct 30 20:36:00.400485 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.400392 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2"] Oct 30 20:36:00.401661 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.401640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:00.402104 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.402190 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb7e64-5a1d-4470-9269-79be9505ebe8-tmp-dir\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.402190 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157b8177-38fc-4986-880f-e820ce5621b1-cert\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.402272 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.402272 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.402239 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:00.402367 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.402311 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.902296344 +0000 UTC m=+31.729954682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:00.402367 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-stats-auth\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.402563 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db8922f9-31fe-4afd-b227-313eecf0fcaa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:00.402623 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.402623 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.402721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.402721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.402721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55m2\" (UniqueName: \"kubernetes.io/projected/cd681674-072e-4dff-a369-9d956c32b960-kube-api-access-l55m2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.402721 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-snapshots\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-serving-cert\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c365d189-b04a-4b12-8519-30c074796180-serving-cert\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd681674-072e-4dff-a369-9d956c32b960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.402904 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqdc\" (UniqueName: \"kubernetes.io/projected/c365d189-b04a-4b12-8519-30c074796180-kube-api-access-frqdc\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ngt\" (UniqueName: \"kubernetes.io/projected/8bfb7e64-5a1d-4470-9269-79be9505ebe8-kube-api-access-s2ngt\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.402990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d68f309-a728-4000-a631-86ef2c50b3e0-serving-cert\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqr4\" (UniqueName: \"kubernetes.io/projected/3d68f309-a728-4000-a631-86ef2c50b3e0-kube-api-access-7hqr4\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.403065 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.403130 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.90311144 +0000 UTC m=+31.730769778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : secret "router-metrics-certs-default" not found Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktzm\" (UniqueName: \"kubernetes.io/projected/470aba85-18f4-4acc-9e98-6233c2ae3c9c-kube-api-access-4ktzm\") pod \"kas-connection-checker-685895c7c9-sg6jp\" (UID: \"470aba85-18f4-4acc-9e98-6233c2ae3c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:00.403257 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-service-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd681674-072e-4dff-a369-9d956c32b960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-tmp\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-default-certificate\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcp7p\" (UniqueName: \"kubernetes.io/projected/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-kube-api-access-fcp7p\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-config\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-config\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea62097-316d-49f5-bccf-dba48029bb6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskcx\" (UniqueName: \"kubernetes.io/projected/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab-kube-api-access-hskcx\") pod \"network-check-source-6f78d59f99-v2p5n\" (UID: \"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qrj\" (UniqueName: \"kubernetes.io/projected/f340b19f-a26a-4514-8785-f0714fc17696-kube-api-access-j5qrj\") pod \"volume-data-source-validator-58c9f5f46d-7sjnz\" (UID: \"f340b19f-a26a-4514-8785-f0714fc17696\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.403986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdngs\" (UniqueName: \"kubernetes.io/projected/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-kube-api-access-sdngs\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6xs\" (UniqueName: \"kubernetes.io/projected/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-kube-api-access-bv6xs\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhrc\" (UniqueName: \"kubernetes.io/projected/7ea62097-316d-49f5-bccf-dba48029bb6d-kube-api-access-pjhrc\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.403948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfb7e64-5a1d-4470-9269-79be9505ebe8-config-volume\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.404688 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.404006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5jz\" (UniqueName: \"kubernetes.io/projected/157b8177-38fc-4986-880f-e820ce5621b1-kube-api-access-sg5jz\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.405005 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.404936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-service-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.405487 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.405467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd681674-072e-4dff-a369-9d956c32b960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.405717 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.405690 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.405812 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.405746 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-tmp\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.405875 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.405852 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:00.405946 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.405924 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-serving-cert\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.405998 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.405953 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.905937429 +0000 UTC m=+31.733595745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:00.406097 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.406074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.406916 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.406769 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db8922f9-31fe-4afd-b227-313eecf0fcaa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:00.406916 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.406880 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:00.406916 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.406894 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:00.407140 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.406949 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.906933575 +0000 UTC m=+31.734591908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:00.407228 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.407205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.407852 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.407832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.408994 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.407957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea62097-316d-49f5-bccf-dba48029bb6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.408994 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.408290 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.409227 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.408547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.409309 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.409264 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.409383 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.409348 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-config\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.409734 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.409701 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.410013 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.409989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-config\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.410178 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.410157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.410307 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.410284 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c365d189-b04a-4b12-8519-30c074796180-serving-cert\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.410433 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.410365 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.410538 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.410522 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.910508358 +0000 UTC m=+31.738166693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : configmap references non-existent config key: service-ca.crt Oct 30 20:36:00.410783 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.410768 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:00.410828 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.410787 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:00.410866 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.410829 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.910813966 +0000 UTC m=+31.738472285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:00.410942 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.410926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.411017 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.411000 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 20:36:00.411124 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.411024 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-snapshots\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.411124 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.411048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert podName:db8922f9-31fe-4afd-b227-313eecf0fcaa nodeName:}" failed. No retries permitted until 2026-10-30 20:36:00.911032203 +0000 UTC m=+31.738690517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zbpwc" (UID: "db8922f9-31fe-4afd-b227-313eecf0fcaa") : secret "networking-console-plugin-cert" not found Oct 30 20:36:00.411124 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.411056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd681674-072e-4dff-a369-9d956c32b960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.411366 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.411347 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-stats-auth\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.411689 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.411666 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-default-certificate\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.412599 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.412579 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d68f309-a728-4000-a631-86ef2c50b3e0-serving-cert\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.413072 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.413022 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.424471 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.424447 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:36:00.424603 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.424495 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2nx5p\"" Oct 30 20:36:00.437318 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.437298 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:36:00.437463 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.437445 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:36:00.455700 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.455674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:00.463893 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.463812 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85df8d78bd-v8dx4"] Oct 30 20:36:00.469572 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.469516 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktzm\" (UniqueName: \"kubernetes.io/projected/470aba85-18f4-4acc-9e98-6233c2ae3c9c-kube-api-access-4ktzm\") pod \"kas-connection-checker-685895c7c9-sg6jp\" (UID: \"470aba85-18f4-4acc-9e98-6233c2ae3c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:00.474033 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.473967 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcp7p\" (UniqueName: \"kubernetes.io/projected/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-kube-api-access-fcp7p\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.474170 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.474059 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qrj\" (UniqueName: \"kubernetes.io/projected/f340b19f-a26a-4514-8785-f0714fc17696-kube-api-access-j5qrj\") pod \"volume-data-source-validator-58c9f5f46d-7sjnz\" (UID: \"f340b19f-a26a-4514-8785-f0714fc17696\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:00.474170 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.474158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqdc\" (UniqueName: \"kubernetes.io/projected/c365d189-b04a-4b12-8519-30c074796180-kube-api-access-frqdc\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.474594 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.474563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.476037 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.476012 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqr4\" (UniqueName: \"kubernetes.io/projected/3d68f309-a728-4000-a631-86ef2c50b3e0-kube-api-access-7hqr4\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.477133 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.477101 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.477218 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.477158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:00.478130 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.478111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.478596 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.478567 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55m2\" (UniqueName: \"kubernetes.io/projected/cd681674-072e-4dff-a369-9d956c32b960-kube-api-access-l55m2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.478674 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.478600 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhrc\" (UniqueName: \"kubernetes.io/projected/7ea62097-316d-49f5-bccf-dba48029bb6d-kube-api-access-pjhrc\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.478743 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.478679 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6xs\" (UniqueName: \"kubernetes.io/projected/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-kube-api-access-bv6xs\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.483478 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.483400 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdngs\" (UniqueName: \"kubernetes.io/projected/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-kube-api-access-sdngs\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.486374 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.486341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:00.505156 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5jz\" (UniqueName: \"kubernetes.io/projected/157b8177-38fc-4986-880f-e820ce5621b1-kube-api-access-sg5jz\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.505324 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb7e64-5a1d-4470-9269-79be9505ebe8-tmp-dir\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.505324 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157b8177-38fc-4986-880f-e820ce5621b1-cert\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.505612 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ngt\" (UniqueName: \"kubernetes.io/projected/8bfb7e64-5a1d-4470-9269-79be9505ebe8-kube-api-access-s2ngt\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.506030 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505787 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb7e64-5a1d-4470-9269-79be9505ebe8-tmp-dir\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.506030 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskcx\" (UniqueName: \"kubernetes.io/projected/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab-kube-api-access-hskcx\") pod \"network-check-source-6f78d59f99-v2p5n\" (UID: \"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:00.506030 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.505976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.506220 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.506045 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:00.506220 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.506088 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:01.006075714 +0000 UTC m=+31.833734029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:00.506220 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.506153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfb7e64-5a1d-4470-9269-79be9505ebe8-config-volume\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.506669 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.506645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfb7e64-5a1d-4470-9269-79be9505ebe8-config-volume\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.508035 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.508015 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157b8177-38fc-4986-880f-e820ce5621b1-cert\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.559259 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.559226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:00.568126 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.567037 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ngt\" (UniqueName: \"kubernetes.io/projected/8bfb7e64-5a1d-4470-9269-79be9505ebe8-kube-api-access-s2ngt\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:00.568126 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.567154 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskcx\" (UniqueName: \"kubernetes.io/projected/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab-kube-api-access-hskcx\") pod \"network-check-source-6f78d59f99-v2p5n\" (UID: \"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:00.568126 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.567151 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5jz\" (UniqueName: \"kubernetes.io/projected/157b8177-38fc-4986-880f-e820ce5621b1-kube-api-access-sg5jz\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.583572 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.583545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:00.603619 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.603572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:00.710698 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.710616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:00.764362 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.764336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:00.773494 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.773462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:00.775588 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.775565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:00.863459 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.863161 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tkhfn"] Oct 30 20:36:00.870718 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:00.870685 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd9cad_031b_4a85_a86d_a04eecb18d95.slice/crio-8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88 WatchSource:0}: Error finding container 8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88: Status 404 returned error can't find the container with id 8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88 Oct 30 20:36:00.908981 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.908952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:00.909093 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.909015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:00.909157 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.909096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:00.909157 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.909120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:00.909262 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909249 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 20:36:00.909312 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909294 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:01.909282663 +0000 UTC m=+32.736940978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : secret "router-metrics-certs-default" not found Oct 30 20:36:00.909373 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909340 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:00.909373 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909348 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:00.909373 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909369 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:01.90936326 +0000 UTC m=+32.737021576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:00.909558 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909403 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:00.909558 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909446 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:01.909437316 +0000 UTC m=+32.737095631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:00.909558 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909500 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:00.909558 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:00.909532 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:01.909522492 +0000 UTC m=+32.737180813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:00.928303 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.928268 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldcc4"] Oct 30 20:36:00.932197 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:00.932166 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d94d6f_863f_4819_b36f_5d2c753471bd.slice/crio-0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878 WatchSource:0}: Error finding container 0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878: Status 404 returned error can't find the container with id 0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878 Oct 30 20:36:00.978222 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.978186 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv"] Oct 30 20:36:00.983003 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:00.982981 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd681674_072e_4dff_a369_9d956c32b960.slice/crio-77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968 WatchSource:0}: Error finding container 77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968: Status 404 returned error can't find the container with id 77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968 Oct 30 20:36:00.990090 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:00.989882 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2lsbc"] Oct 30 20:36:00.995114 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:00.995080 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d68f309_a728_4000_a631_86ef2c50b3e0.slice/crio-62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b WatchSource:0}: Error finding container 62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b: Status 404 returned error can't find the container with id 62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b Oct 30 20:36:01.010455 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.010435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:01.010535 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.010516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:01.010580 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.010549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:01.010580 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.010569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:01.010646 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010580 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:01.010646 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010630 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:02.010616847 +0000 UTC m=+32.838275162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:01.010708 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010666 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:02.01065064 +0000 UTC m=+32.838308960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : configmap references non-existent config key: service-ca.crt Oct 30 20:36:01.010708 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010684 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:01.010708 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010698 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:01.010795 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010730 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:02.010721225 +0000 UTC m=+32.838379544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:01.010795 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010683 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 20:36:01.010795 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.010775 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert podName:db8922f9-31fe-4afd-b227-313eecf0fcaa nodeName:}" failed. No retries permitted until 2026-10-30 20:36:02.010766629 +0000 UTC m=+32.838424944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zbpwc" (UID: "db8922f9-31fe-4afd-b227-313eecf0fcaa") : secret "networking-console-plugin-cert" not found Oct 30 20:36:01.024102 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.024079 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz"] Oct 30 20:36:01.029494 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:01.029468 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf340b19f_a26a_4514_8785_f0714fc17696.slice/crio-7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08 WatchSource:0}: Error finding container 7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08: Status 404 returned error can't find the container with id 7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08 Oct 30 20:36:01.076282 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.076248 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp"] Oct 30 20:36:01.080129 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:01.080100 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3af4a4_5731_41d0_8f14_cbe1543de1f7.slice/crio-433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7 WatchSource:0}: Error finding container 433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7: Status 404 returned error can't find the container with id 433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7 Oct 30 20:36:01.132046 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.132017 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n"] Oct 30 20:36:01.135613 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:01.135583 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b81cd8d_9d74_4e6d_bda5_7b26aa16feab.slice/crio-59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6 WatchSource:0}: Error finding container 59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6: Status 404 returned error can't find the container with id 59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6 Oct 30 20:36:01.175136 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.175103 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-sg6jp"] Oct 30 20:36:01.177921 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:01.177896 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470aba85_18f4_4acc_9e98_6233c2ae3c9c.slice/crio-7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d WatchSource:0}: Error finding container 7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d: Status 404 returned error can't find the container with id 7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d Oct 30 20:36:01.199196 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.199168 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tmxhb"] Oct 30 20:36:01.202841 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:01.202812 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc365d189_b04a_4b12_8519_30c074796180.slice/crio-f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca WatchSource:0}: Error finding container f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca: Status 404 returned error can't find the container with id f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca Oct 30 20:36:01.236353 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.236278 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6pl6"] Oct 30 20:36:01.239293 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:01.239268 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157b8177_38fc_4986_880f_e820ce5621b1.slice/crio-37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96 WatchSource:0}: Error finding container 37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96: Status 404 returned error can't find the container with id 37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96 Oct 30 20:36:01.437993 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.437951 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:01.641776 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.641749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:01.641776 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.641774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:01.642007 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.641913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:01.669753 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.669721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:01.695879 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.695853 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x2fjq\"" Oct 30 20:36:01.696015 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.695860 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:36:01.696091 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.696050 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:36:01.696147 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.696131 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hqxsv\"" Oct 30 20:36:01.696676 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.696657 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-csvtm\"" Oct 30 20:36:01.859294 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.859264 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" event={"ID":"cd681674-072e-4dff-a369-9d956c32b960","Type":"ContainerStarted","Data":"77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968"} Oct 30 20:36:01.860790 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.860745 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" event={"ID":"70d94d6f-863f-4819-b36f-5d2c753471bd","Type":"ContainerStarted","Data":"5cda59af193d64ce8c109f59f40563641a545276d7b769b895cfd8ed7643c519"} Oct 30 20:36:01.860790 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.860784 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" event={"ID":"70d94d6f-863f-4819-b36f-5d2c753471bd","Type":"ContainerStarted","Data":"0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878"} Oct 30 20:36:01.862169 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.862145 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" event={"ID":"470aba85-18f4-4acc-9e98-6233c2ae3c9c","Type":"ContainerStarted","Data":"fb87908068998150204ed00dd0b05571692e5ef13aff11a35fae02550eddf478"} Oct 30 20:36:01.862274 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.862188 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" event={"ID":"470aba85-18f4-4acc-9e98-6233c2ae3c9c","Type":"ContainerStarted","Data":"7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d"} Oct 30 20:36:01.863203 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.863186 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" event={"ID":"3d68f309-a728-4000-a631-86ef2c50b3e0","Type":"ContainerStarted","Data":"62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b"} Oct 30 20:36:01.864254 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.864235 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6pl6" event={"ID":"157b8177-38fc-4986-880f-e820ce5621b1","Type":"ContainerStarted","Data":"37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96"} Oct 30 20:36:01.865590 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.865562 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" event={"ID":"cacd9cad-031b-4a85-a86d-a04eecb18d95","Type":"ContainerStarted","Data":"fd36d21fe04ef497a14e04350e222caed03ca16743c3fd3d698600c387e2e8fa"} Oct 30 20:36:01.865671 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.865599 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" event={"ID":"cacd9cad-031b-4a85-a86d-a04eecb18d95","Type":"ContainerStarted","Data":"8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88"} Oct 30 20:36:01.866663 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.866646 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" event={"ID":"f340b19f-a26a-4514-8785-f0714fc17696","Type":"ContainerStarted","Data":"7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08"} Oct 30 20:36:01.867592 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.867575 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" event={"ID":"0a3af4a4-5731-41d0-8f14-cbe1543de1f7","Type":"ContainerStarted","Data":"433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7"} Oct 30 20:36:01.868509 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.868490 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" event={"ID":"c365d189-b04a-4b12-8519-30c074796180","Type":"ContainerStarted","Data":"f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca"} Oct 30 20:36:01.869436 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.869402 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" event={"ID":"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab","Type":"ContainerStarted","Data":"59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6"} Oct 30 20:36:01.893351 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.893286 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:01.918361 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.918323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:01.918512 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.918374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:01.918512 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918488 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:01.918512 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918505 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.918513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918535 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918552 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.918538436 +0000 UTC m=+34.746196751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:01.918570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918587 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.91857188 +0000 UTC m=+34.746230195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918611 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918613 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.918631081 +0000 UTC m=+34.746289419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : secret "router-metrics-certs-default" not found Oct 30 20:36:01.918680 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:01.918658 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:03.918644479 +0000 UTC m=+34.746302795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:02.019614 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.019584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:02.019791 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.019629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:02.019791 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.019650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:02.019791 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.019749 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 20:36:02.019791 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.019755 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:02.019791 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.019775 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:02.020049 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.019775 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:04.019755176 +0000 UTC m=+34.847413520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : configmap references non-existent config key: service-ca.crt Oct 30 20:36:02.020049 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.019863 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert podName:db8922f9-31fe-4afd-b227-313eecf0fcaa nodeName:}" failed. No retries permitted until 2026-10-30 20:36:04.019846142 +0000 UTC m=+34.847504465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zbpwc" (UID: "db8922f9-31fe-4afd-b227-313eecf0fcaa") : secret "networking-console-plugin-cert" not found Oct 30 20:36:02.020049 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.019887 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:04.019880954 +0000 UTC m=+34.847539269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:02.020049 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.019946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:02.020255 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.020074 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:02.020255 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:02.020117 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:04.02010695 +0000 UTC m=+34.847765279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:02.272724 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.272162 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" podStartSLOduration=384.272144211 podStartE2EDuration="6m24.272144211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:02.27180373 +0000 UTC m=+33.099462080" watchObservedRunningTime="2026-10-30 20:36:02.272144211 +0000 UTC m=+33.099802550" Oct 30 20:36:02.401953 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.401306 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" podStartSLOduration=384.401286813 podStartE2EDuration="6m24.401286813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:02.400066777 +0000 UTC m=+33.227725115" watchObservedRunningTime="2026-10-30 20:36:02.401286813 +0000 UTC m=+33.228945152" Oct 30 20:36:02.515287 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:02.515222 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" podStartSLOduration=384.515201499 podStartE2EDuration="6m24.515201499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:02.513907751 +0000 UTC m=+33.341566088" watchObservedRunningTime="2026-10-30 20:36:02.515201499 +0000 UTC m=+33.342859839" Oct 30 20:36:03.449342 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.449136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:03.449834 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.449403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:03.449834 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.449311 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 20:36:03.449834 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.449587 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:35.449571308 +0000 UTC m=+66.277229641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : secret "metrics-daemon-secret" not found Oct 30 20:36:03.461324 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.461204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:03.501359 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.501168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:03.550757 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.550654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:03.555084 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.555027 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:03.766027 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.765760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:03.885986 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.885947 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" event={"ID":"f340b19f-a26a-4514-8785-f0714fc17696","Type":"ContainerStarted","Data":"7f0882c070b46e40e296835fb027eeb0a916caa3cdc3268df0d6ea9d0f07cd63"} Oct 30 20:36:03.926036 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.925952 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2kjtc"] Oct 30 20:36:03.937636 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:03.937605 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed71fea_5f6c_4ba4_be09_b524c1abfc4a.slice/crio-8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d WatchSource:0}: Error finding container 8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d: Status 404 returned error can't find the container with id 8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.955179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.955231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.955345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955354 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955376 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:03.955381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955450 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:07.955430197 +0000 UTC m=+38.783088517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955513 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955570 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:07.955554931 +0000 UTC m=+38.783213262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : secret "router-metrics-certs-default" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955606 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955638 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955644 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:07.955631791 +0000 UTC m=+38.783290108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:03.955694 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:03.955675 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:07.955663664 +0000 UTC m=+38.783321983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:04.004890 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.004728 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" podStartSLOduration=349.388358419 podStartE2EDuration="5m52.004705105s" totalImagesPullingTime="2.616346686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:04.003050652 +0000 UTC m=+34.830708990" watchObservedRunningTime="2026-10-30 20:36:04.004705105 +0000 UTC m=+34.832363446" Oct 30 20:36:04.056564 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.056528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:04.056832 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.056583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:04.056832 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.056714 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:04.056832 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.056737 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:04.056832 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.056788 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:08.056767309 +0000 UTC m=+38.884425641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : configmap references non-existent config key: service-ca.crt Oct 30 20:36:04.057128 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.056846 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:08.056804193 +0000 UTC m=+38.884462514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:04.057128 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.056884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:04.057128 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.057005 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 20:36:04.057128 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.057009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:04.057128 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.057054 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert podName:db8922f9-31fe-4afd-b227-313eecf0fcaa nodeName:}" failed. No retries permitted until 2026-10-30 20:36:08.057035763 +0000 UTC m=+38.884694078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zbpwc" (UID: "db8922f9-31fe-4afd-b227-313eecf0fcaa") : secret "networking-console-plugin-cert" not found Oct 30 20:36:04.057128 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.057120 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:04.057467 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:04.057163 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:08.057151921 +0000 UTC m=+38.884810244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:04.115446 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.115374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:04.236218 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.236182 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jdq2q"] Oct 30 20:36:04.241805 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:04.241759 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f48b68d_7d92_4417_915f_785bb3e457ab.slice/crio-7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b WatchSource:0}: Error finding container 7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b: Status 404 returned error can't find the container with id 7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b Oct 30 20:36:04.261971 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.261854 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:04.889784 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.889743 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jdq2q" event={"ID":"5f48b68d-7d92-4417-915f-785bb3e457ab","Type":"ContainerStarted","Data":"7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b"} Oct 30 20:36:04.891022 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:04.890995 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2kjtc" event={"ID":"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a","Type":"ContainerStarted","Data":"8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d"} Oct 30 20:36:05.806357 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:05.806299 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:05.897443 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:05.897392 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/0.log" Oct 30 20:36:05.897897 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:05.897596 2577 generic.go:356] "Generic (PLEG): container finished" podID="3d68f309-a728-4000-a631-86ef2c50b3e0" containerID="12d4f1703cf33d37bccdd35d4c711c9c71063ca44881c7b517a023e236c2e8c6" exitCode=255 Oct 30 20:36:05.897897 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:05.897654 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" event={"ID":"3d68f309-a728-4000-a631-86ef2c50b3e0","Type":"ContainerDied","Data":"12d4f1703cf33d37bccdd35d4c711c9c71063ca44881c7b517a023e236c2e8c6"} Oct 30 20:36:05.898028 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:05.898000 2577 scope.go:122] "RemoveContainer" containerID="12d4f1703cf33d37bccdd35d4c711c9c71063ca44881c7b517a023e236c2e8c6" Oct 30 20:36:06.006084 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.006052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:06.904547 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.904502 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" event={"ID":"cd681674-072e-4dff-a369-9d956c32b960","Type":"ContainerStarted","Data":"315e65897483b7b8d176bf98cfd0e9b88a3e310223d32046e39511f5308c4211"} Oct 30 20:36:06.908426 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.908386 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:36:06.909866 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.908887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/0.log" Oct 30 20:36:06.909866 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.908929 2577 generic.go:356] "Generic (PLEG): container finished" podID="3d68f309-a728-4000-a631-86ef2c50b3e0" containerID="48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557" exitCode=255 Oct 30 20:36:06.909866 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.908998 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" event={"ID":"3d68f309-a728-4000-a631-86ef2c50b3e0","Type":"ContainerDied","Data":"48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557"} Oct 30 20:36:06.909866 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.909044 2577 scope.go:122] "RemoveContainer" containerID="12d4f1703cf33d37bccdd35d4c711c9c71063ca44881c7b517a023e236c2e8c6" Oct 30 20:36:06.909866 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:06.909452 2577 scope.go:122] "RemoveContainer" containerID="48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557" Oct 30 20:36:06.909866 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:06.909634 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2lsbc_openshift-console-operator(3d68f309-a728-4000-a631-86ef2c50b3e0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" podUID="3d68f309-a728-4000-a631-86ef2c50b3e0" Oct 30 20:36:07.022552 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.022487 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" podStartSLOduration=349.816373649 podStartE2EDuration="5m55.022469715s" totalImagesPullingTime="5.206096066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:07.022339753 +0000 UTC m=+37.849998090" watchObservedRunningTime="2026-10-30 20:36:07.022469715 +0000 UTC m=+37.850128054" Oct 30 20:36:07.909906 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.909705 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:07.915213 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.915168 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6pl6" event={"ID":"157b8177-38fc-4986-880f-e820ce5621b1","Type":"ContainerStarted","Data":"7b45ddd397f7bcc4d66447dc14183b96673e0b9212ead6b6b04b4c4f97ce7c98"} Oct 30 20:36:07.918077 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.918047 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" event={"ID":"0a3af4a4-5731-41d0-8f14-cbe1543de1f7","Type":"ContainerStarted","Data":"9060c5c33ce9c0d35afc45658825b75fc2a44cb53eca60df3681ee64418676c4"} Oct 30 20:36:07.920906 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.920873 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" event={"ID":"c365d189-b04a-4b12-8519-30c074796180","Type":"ContainerStarted","Data":"e0e26d7453e756e3ec10d76eca29ed2b2b51047c590ebcca688a17be73d87b77"} Oct 30 20:36:07.924086 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.924061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:36:07.924782 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.924754 2577 scope.go:122] "RemoveContainer" containerID="48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557" Oct 30 20:36:07.925011 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:07.924989 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2lsbc_openshift-console-operator(3d68f309-a728-4000-a631-86ef2c50b3e0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" podUID="3d68f309-a728-4000-a631-86ef2c50b3e0" Oct 30 20:36:07.999289 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.998639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:07.999289 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.998704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:07.999289 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.998928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:07.999289 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:07.999015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:07.999611 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:07.999684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:15.99966081 +0000 UTC m=+46.827319126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : secret "router-metrics-certs-default" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000066 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000101 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000119 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000124 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.000109482 +0000 UTC m=+46.827767823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000165 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.000149754 +0000 UTC m=+46.827808095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000360 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:08.000465 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.000432 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.0004003 +0000 UTC m=+46.828058642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:08.047381 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.047127 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s6pl6" podStartSLOduration=1.684951504 podStartE2EDuration="8.047108082s" totalImagesPullingTime="6.362156578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:08.04609872 +0000 UTC m=+38.873757059" watchObservedRunningTime="2026-10-30 20:36:08.047108082 +0000 UTC m=+38.874766423" Oct 30 20:36:08.078492 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.078401 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:08.099596 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.099565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:08.099815 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.099671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:08.099815 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.099715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:08.099815 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099733 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:08.099815 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.099742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099845 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.099827121 +0000 UTC m=+46.927485442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099848 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099863 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099902 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.099892916 +0000 UTC m=+46.927551233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099916 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.09990807 +0000 UTC m=+46.927566390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : configmap references non-existent config key: service-ca.crt Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099956 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 20:36:08.100016 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.099998 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert podName:db8922f9-31fe-4afd-b227-313eecf0fcaa nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.099981162 +0000 UTC m=+46.927639483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zbpwc" (UID: "db8922f9-31fe-4afd-b227-313eecf0fcaa") : secret "networking-console-plugin-cert" not found Oct 30 20:36:08.131069 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.131038 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx"] Oct 30 20:36:08.143246 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.142879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:08.162341 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.162244 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h"] Oct 30 20:36:08.165307 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.165070 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" podStartSLOduration=350.028659495 podStartE2EDuration="5m56.165057598s" totalImagesPullingTime="6.136398103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:08.164015033 +0000 UTC m=+38.991673375" watchObservedRunningTime="2026-10-30 20:36:08.165057598 +0000 UTC m=+38.992715932" Oct 30 20:36:08.173522 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.173025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:08.192224 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.192198 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx"] Oct 30 20:36:08.195542 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.195467 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 20:36:08.195542 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.195468 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:08.195715 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.195652 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v74vw\"" Oct 30 20:36:08.200799 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.200285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmm7\" (UniqueName: \"kubernetes.io/projected/f23150de-f251-49ea-a39c-9517da973316-kube-api-access-vhmm7\") pod \"migrator-66df5d496b-psb5h\" (UID: \"f23150de-f251-49ea-a39c-9517da973316\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:08.200799 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.200620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69v9h\" (UniqueName: \"kubernetes.io/projected/ee569c03-5200-472a-8348-382a2da0327b-kube-api-access-69v9h\") pod \"migrator-66df5d496b-8fmqx\" (UID: \"ee569c03-5200-472a-8348-382a2da0327b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:08.241881 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.241842 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h"] Oct 30 20:36:08.254741 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.254706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:08.298502 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.298439 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" podStartSLOduration=350.28518336 podStartE2EDuration="5m56.298405427s" totalImagesPullingTime="6.013222067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:08.297424325 +0000 UTC m=+39.125082659" watchObservedRunningTime="2026-10-30 20:36:08.298405427 +0000 UTC m=+39.126063767" Oct 30 20:36:08.301364 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.301331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v9h\" (UniqueName: \"kubernetes.io/projected/ee569c03-5200-472a-8348-382a2da0327b-kube-api-access-69v9h\") pod \"migrator-66df5d496b-8fmqx\" (UID: \"ee569c03-5200-472a-8348-382a2da0327b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:08.301514 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.301405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmm7\" (UniqueName: \"kubernetes.io/projected/f23150de-f251-49ea-a39c-9517da973316-kube-api-access-vhmm7\") pod \"migrator-66df5d496b-psb5h\" (UID: \"f23150de-f251-49ea-a39c-9517da973316\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:08.365793 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.365735 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v9h\" (UniqueName: \"kubernetes.io/projected/ee569c03-5200-472a-8348-382a2da0327b-kube-api-access-69v9h\") pod \"migrator-66df5d496b-8fmqx\" (UID: \"ee569c03-5200-472a-8348-382a2da0327b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:08.370839 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.370498 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmm7\" (UniqueName: \"kubernetes.io/projected/f23150de-f251-49ea-a39c-9517da973316-kube-api-access-vhmm7\") pod \"migrator-66df5d496b-psb5h\" (UID: \"f23150de-f251-49ea-a39c-9517da973316\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:08.410884 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.410851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:08.465814 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.465782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:08.500764 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.500725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:08.744200 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.744123 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bdlwf"] Oct 30 20:36:08.770601 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.770569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.807374 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.807037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aedd749-52bf-473f-86bb-f42e2cd459d0-crio-socket\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.807374 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.807141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.807374 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.807186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxnr\" (UniqueName: \"kubernetes.io/projected/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-api-access-tnxnr\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.807374 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.807256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aedd749-52bf-473f-86bb-f42e2cd459d0-data-volume\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.807374 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.807284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.808232 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.807713 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdlwf"] Oct 30 20:36:08.862558 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.862511 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:36:08.876074 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.876023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:08.877117 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.877087 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx"] Oct 30 20:36:08.881979 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:08.881953 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee569c03_5200_472a_8348_382a2da0327b.slice/crio-7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd WatchSource:0}: Error finding container 7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd: Status 404 returned error can't find the container with id 7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd Oct 30 20:36:08.882543 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.882524 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:36:08.882806 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.882790 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mk8vz\"" Oct 30 20:36:08.909095 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aedd749-52bf-473f-86bb-f42e2cd459d0-crio-socket\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.909266 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.909266 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxnr\" (UniqueName: \"kubernetes.io/projected/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-api-access-tnxnr\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.909266 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aedd749-52bf-473f-86bb-f42e2cd459d0-data-volume\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.909266 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.909255 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:36:08.909472 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.909321 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:08.909472 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.909330 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:09.409309046 +0000 UTC m=+40.236967363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : configmap "kube-rbac-proxy" not found Oct 30 20:36:08.909472 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909326 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aedd749-52bf-473f-86bb-f42e2cd459d0-crio-socket\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.909472 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.909472 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:08.909367 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:09.409353083 +0000 UTC m=+40.237011419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:08.909767 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.909602 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aedd749-52bf-473f-86bb-f42e2cd459d0-data-volume\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.922002 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.921968 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h"] Oct 30 20:36:08.925260 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:08.925233 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23150de_f251_49ea_a39c_9517da973316.slice/crio-5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787 WatchSource:0}: Error finding container 5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787: Status 404 returned error can't find the container with id 5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787 Oct 30 20:36:08.929903 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.929879 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" event={"ID":"ee569c03-5200-472a-8348-382a2da0327b","Type":"ContainerStarted","Data":"7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd"} Oct 30 20:36:08.931670 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.931641 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" event={"ID":"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab","Type":"ContainerStarted","Data":"60107d1c09eb0ae904ebc81a441519a81082dcf0d709705a8bdd95d9af62aa6f"} Oct 30 20:36:08.933177 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.933154 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jdq2q" event={"ID":"5f48b68d-7d92-4417-915f-785bb3e457ab","Type":"ContainerStarted","Data":"1193776d462afb0c59efc63c1355b453ff44d9bf7b7ed89d8437f7ced68f57b2"} Oct 30 20:36:08.933263 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.933245 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:08.973468 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.973434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxnr\" (UniqueName: \"kubernetes.io/projected/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-api-access-tnxnr\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:08.998500 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:08.998393 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jfr7m-mw5r6"] Oct 30 20:36:09.022357 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.022318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.029322 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.029034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:09.048849 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.048618 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" podStartSLOduration=348.219393859 podStartE2EDuration="5m55.048599454s" totalImagesPullingTime="6.829205595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:09.046821131 +0000 UTC m=+39.874479467" watchObservedRunningTime="2026-10-30 20:36:09.048599454 +0000 UTC m=+39.876257794" Oct 30 20:36:09.059872 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.059835 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jfr7m-mw5r6"] Oct 30 20:36:09.112109 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.112077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.112309 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.112137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.112378 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.112318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.112582 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.112384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.163249 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.163187 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jdq2q" podStartSLOduration=35.076800394 podStartE2EDuration="39.163165078s" totalImagesPullingTime="4.086364684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:09.162530037 +0000 UTC m=+39.990188375" watchObservedRunningTime="2026-10-30 20:36:09.163165078 +0000 UTC m=+39.990823416" Oct 30 20:36:09.213236 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.213195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.213484 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.213459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.213614 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.213522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.213686 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.213635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.213686 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.213642 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.214105 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.214087 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.214452 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.214400 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.216115 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.216088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:09.275041 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.274968 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.353002 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.352968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:09.372312 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.372268 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:09.416589 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.415803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:09.416589 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.415940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:09.416589 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:09.416069 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:09.416589 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:09.416130 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:10.416113167 +0000 UTC m=+41.243771496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:09.416589 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:09.416507 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:36:09.416589 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:09.416562 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:10.416541198 +0000 UTC m=+41.244199519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : configmap "kube-rbac-proxy" not found Oct 30 20:36:09.748494 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.748401 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jfr7m-mw5r6"] Oct 30 20:36:09.752738 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:09.752699 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99eaa580_8ff6_4a47_971b_80133022ccf0.slice/crio-6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08 WatchSource:0}: Error finding container 6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08: Status 404 returned error can't find the container with id 6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08 Oct 30 20:36:09.939459 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.939399 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" event={"ID":"f23150de-f251-49ea-a39c-9517da973316","Type":"ContainerStarted","Data":"5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787"} Oct 30 20:36:09.941643 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.941618 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" event={"ID":"99eaa580-8ff6-4a47-971b-80133022ccf0","Type":"ContainerStarted","Data":"6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690"} Oct 30 20:36:09.941783 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:09.941664 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" event={"ID":"99eaa580-8ff6-4a47-971b-80133022ccf0","Type":"ContainerStarted","Data":"6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08"} Oct 30 20:36:10.065919 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.065816 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" podStartSLOduration=2.065796282 podStartE2EDuration="2.065796282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:10.061523282 +0000 UTC m=+40.889181620" watchObservedRunningTime="2026-10-30 20:36:10.065796282 +0000 UTC m=+40.893454620" Oct 30 20:36:10.066821 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.066797 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h4ncf"] Oct 30 20:36:10.086009 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.085985 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.125347 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.125315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtmj\" (UniqueName: \"kubernetes.io/projected/ed47a78a-0c03-4f46-b274-5b1d162d1c73-kube-api-access-cvtmj\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.125549 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.125439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-cabundle\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.125549 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.125503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-key\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.128330 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.128250 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h4ncf"] Oct 30 20:36:10.138661 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.138368 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qdrm5\"" Oct 30 20:36:10.138661 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.138499 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 20:36:10.138661 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.138601 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 20:36:10.138883 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.138734 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 20:36:10.138883 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.138762 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 20:36:10.226910 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.226779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-key\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.227095 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.227048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtmj\" (UniqueName: \"kubernetes.io/projected/ed47a78a-0c03-4f46-b274-5b1d162d1c73-kube-api-access-cvtmj\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.227192 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.227136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-cabundle\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.228253 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.228077 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-cabundle\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.230230 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.230201 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-key\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.288740 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.288697 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtmj\" (UniqueName: \"kubernetes.io/projected/ed47a78a-0c03-4f46-b274-5b1d162d1c73-kube-api-access-cvtmj\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.317029 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.316895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:10.413107 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.413076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:10.428899 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.428867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:10.429033 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:10.429012 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:36:10.429096 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.429018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:10.429150 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:10.429088 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:12.429068842 +0000 UTC m=+43.256727172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : configmap "kube-rbac-proxy" not found Oct 30 20:36:10.429150 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:10.429123 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:10.429262 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:10.429174 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:12.4291595 +0000 UTC m=+43.256817832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:10.464898 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.464865 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:10.584700 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.584582 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:10.584700 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.584668 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:10.585160 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.585137 2577 scope.go:122] "RemoveContainer" containerID="48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557" Oct 30 20:36:10.585384 ip-10-0-141-115 kubenswrapper[2577]: E1030 20:36:10.585362 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2lsbc_openshift-console-operator(3d68f309-a728-4000-a631-86ef2c50b3e0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" podUID="3d68f309-a728-4000-a631-86ef2c50b3e0" Oct 30 20:36:10.806429 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.806387 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h4ncf"] Oct 30 20:36:10.810735 ip-10-0-141-115 kubenswrapper[2577]: W1030 20:36:10.810697 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded47a78a_0c03_4f46_b274_5b1d162d1c73.slice/crio-55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d WatchSource:0}: Error finding container 55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d: Status 404 returned error can't find the container with id 55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d Oct 30 20:36:10.947440 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.947384 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" event={"ID":"ed47a78a-0c03-4f46-b274-5b1d162d1c73","Type":"ContainerStarted","Data":"b3ac09abdef06ff2b8f0150319aafc7633334d6c6bd2953c4f4a1f59ea9c1445"} Oct 30 20:36:10.947977 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:10.947469 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" event={"ID":"ed47a78a-0c03-4f46-b274-5b1d162d1c73","Type":"ContainerStarted","Data":"55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d"} Oct 30 20:36:11.086365 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:11.086331 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:11.842343 ip-10-0-141-115 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 20:36:11.842609 ip-10-0-141-115 kubenswrapper[2577]: I1030 20:36:11.842188 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:36:11.853922 ip-10-0-141-115 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 20:36:11.854144 ip-10-0-141-115 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 20:36:11.854308 ip-10-0-141-115 systemd[1]: kubelet.service: Consumed 3.281s CPU time. Oct 30 20:36:11.862077 ip-10-0-141-115 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:36:11.922137 ip-10-0-141-115 kubenswrapper[4988]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:36:11.922137 ip-10-0-141-115 kubenswrapper[4988]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:36:11.922137 ip-10-0-141-115 kubenswrapper[4988]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:36:11.922137 ip-10-0-141-115 kubenswrapper[4988]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:36:11.924253 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924169 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.924253 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924253 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.924253 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924257 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924260 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924263 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924266 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924269 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924271 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924274 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924277 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924279 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924283 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924287 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924292 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924296 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924301 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924304 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924307 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924309 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924312 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924315 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924324 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.924353 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924327 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924330 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924333 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924336 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924339 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924342 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924344 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924347 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924350 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924353 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924356 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924358 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924362 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924368 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924372 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924375 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924377 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924380 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924383 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.924889 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924385 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924388 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924391 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924393 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924396 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924398 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924401 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924405 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924421 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924427 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924431 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924443 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924446 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924449 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924451 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924454 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924457 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924460 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924463 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.925350 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924465 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924468 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924471 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924473 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924476 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924478 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924481 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924483 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924486 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924488 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924490 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924493 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924496 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924498 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924500 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924503 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924505 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924508 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924511 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924514 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.925838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924516 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924519 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924522 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924525 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924528 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924537 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924542 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924545 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924548 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924551 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924553 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924556 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924559 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924561 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924564 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924567 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924570 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924573 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924575 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.926348 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924578 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924581 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924583 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924586 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924588 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924591 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.924594 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925400 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925421 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925427 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925431 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925434 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925437 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925440 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925443 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925445 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925448 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925451 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925454 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.926830 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925457 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925460 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925463 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925466 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925468 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925471 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925473 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925476 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925478 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925481 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925484 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925486 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925489 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925492 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925494 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925497 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925499 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925502 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925505 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.927305 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925507 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925510 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925512 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925515 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925517 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925520 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925523 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925526 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925529 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925531 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925534 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925537 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925539 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925542 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925545 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925548 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925551 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925553 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925556 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925558 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.927833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925561 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925563 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925566 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925569 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925571 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925574 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925576 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925579 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925582 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925584 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925587 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925589 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925592 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925594 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925597 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925599 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925602 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925604 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925607 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925611 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.928323 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925614 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925616 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925619 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925622 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925625 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925627 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925630 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925633 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925635 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925638 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925640 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925643 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925645 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925648 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925650 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925653 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925655 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925657 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925660 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.928890 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925662 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925666 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925670 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925673 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925676 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925679 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925681 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925684 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925686 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925689 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925691 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925694 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925697 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925699 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925702 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.925704 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926185 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926190 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926193 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.929375 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926196 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926199 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926202 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926206 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926210 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926213 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926216 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926219 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926221 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926224 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926228 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926232 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926235 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926238 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926241 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926245 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926247 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926249 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926252 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.929866 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926255 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926257 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926260 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926262 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926265 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926267 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926270 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926273 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926276 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926279 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926282 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926284 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926287 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926289 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926292 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926295 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926298 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926301 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926304 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926306 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.930333 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926309 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926311 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926314 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926316 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926319 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926321 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926324 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926330 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926333 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926335 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926338 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926340 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926343 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926345 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926349 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926351 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926354 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926357 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926360 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926362 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.930838 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926365 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926368 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926370 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926373 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926375 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926378 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926380 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926382 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926385 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926388 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926391 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926394 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926396 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926399 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926401 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926404 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926420 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926424 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926430 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.931345 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926436 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926440 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926444 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926447 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926450 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926453 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926455 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926458 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926461 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926464 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926466 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926469 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926471 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926474 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926477 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926479 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926482 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926484 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926487 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926489 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.931944 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926492 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926496 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926503 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926506 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.926508 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926607 4988 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926617 4988 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926626 4988 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926632 4988 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926640 4988 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926644 4988 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926649 4988 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926655 4988 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926658 4988 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926661 4988 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926665 4988 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926668 4988 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926671 4988 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926674 4988 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926677 4988 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926681 4988 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926684 4988 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926687 4988 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:36:11.932788 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926691 4988 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926694 4988 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926697 4988 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926701 4988 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926704 4988 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926708 4988 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926711 4988 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926715 4988 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926718 4988 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926721 4988 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926724 4988 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926727 4988 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926731 4988 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926734 4988 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926739 4988 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926742 4988 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926745 4988 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926750 4988 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926753 4988 flags.go:64] FLAG: --enable-server="true" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926758 4988 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926765 4988 flags.go:64] FLAG: --event-burst="100" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926773 4988 flags.go:64] FLAG: --event-qps="50" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926776 4988 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926780 4988 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:36:11.933483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926783 4988 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926787 4988 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926790 4988 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926794 4988 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926797 4988 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926800 4988 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926803 4988 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926806 4988 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926809 4988 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926812 4988 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926815 4988 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926818 4988 flags.go:64] FLAG: --feature-gates="" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926822 4988 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926825 4988 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926828 4988 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926832 4988 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926836 4988 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926839 4988 flags.go:64] FLAG: --help="false" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926842 4988 flags.go:64] FLAG: --hostname-override="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926846 4988 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926849 4988 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926853 4988 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926856 4988 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:36:11.934082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926860 4988 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926863 4988 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926867 4988 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926870 4988 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926873 4988 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926876 4988 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926880 4988 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926883 4988 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926886 4988 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926889 4988 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926892 4988 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926895 4988 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926899 4988 flags.go:64] FLAG: --lock-file="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926902 4988 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926907 4988 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926910 4988 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926916 4988 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926919 4988 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926922 4988 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926925 4988 flags.go:64] FLAG: --logging-format="text" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926928 4988 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926931 4988 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926934 4988 flags.go:64] FLAG: --manifest-url="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926937 4988 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926942 4988 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:36:11.934692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926945 4988 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926948 4988 flags.go:64] FLAG: --max-pods="110" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926951 4988 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926955 4988 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926958 4988 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926961 4988 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926965 4988 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926968 4988 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926971 4988 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926980 4988 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926984 4988 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926987 4988 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926991 4988 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926994 4988 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.926996 4988 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927000 4988 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927003 4988 flags.go:64] FLAG: --port="10250" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927006 4988 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927008 4988 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-014ffddc146390e0f" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927012 4988 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927015 4988 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927018 4988 flags.go:64] FLAG: --register-node="true" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927021 4988 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927025 4988 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927028 4988 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927031 4988 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:36:11.935303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927034 4988 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927038 4988 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927041 4988 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927044 4988 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927047 4988 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927050 4988 flags.go:64] FLAG: --runonce="false" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927053 4988 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927058 4988 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927063 4988 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927067 4988 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927072 4988 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927077 4988 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927082 4988 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927086 4988 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927089 4988 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927094 4988 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927097 4988 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927100 4988 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927103 4988 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927106 4988 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927109 4988 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927115 4988 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927117 4988 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927120 4988 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927126 4988 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:36:11.935949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927129 4988 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927132 4988 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927135 4988 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927138 4988 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927141 4988 flags.go:64] FLAG: --v="2" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927145 4988 flags.go:64] FLAG: --version="false" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927150 4988 flags.go:64] FLAG: --vmodule="" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927154 4988 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927157 4988 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927292 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927296 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927298 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927301 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927304 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927306 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927309 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927313 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927315 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927318 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927321 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927323 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.936729 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927327 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927330 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927333 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927336 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927339 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927341 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927344 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927347 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927350 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927353 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927355 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927358 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927361 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927363 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927366 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927369 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927371 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927374 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927377 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927380 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.937257 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927383 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927386 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927395 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927398 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927402 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927405 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927419 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927422 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927425 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927427 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927430 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927433 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927436 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927444 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927447 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927449 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927452 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927456 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927460 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.937964 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927463 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927467 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927470 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927473 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927475 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927478 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927481 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927484 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927487 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927489 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927492 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927494 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927497 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927500 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927502 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927507 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927509 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927512 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927515 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927518 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.938456 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927528 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927531 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927534 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927537 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927539 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927542 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927546 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927549 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927552 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927555 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927558 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927560 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927563 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927565 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927568 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927570 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927573 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927576 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927578 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.938941 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927581 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927583 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927586 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927589 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927591 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927594 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927597 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927599 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927603 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927605 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927608 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927611 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927613 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927616 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927618 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927621 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.939391 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.927649 4988 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927691 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927696 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927699 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927703 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927707 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927709 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927712 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927714 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927717 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927720 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927722 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927725 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927727 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927730 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927733 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927735 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927738 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927740 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.939818 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927743 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927745 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927748 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927750 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927753 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927756 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927760 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927762 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927765 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927767 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927770 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927772 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927775 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927778 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927780 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927783 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927785 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927790 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927792 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927796 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.940426 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927798 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927801 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927803 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927806 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927808 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927811 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927813 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927816 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927819 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927821 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927824 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927826 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927829 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927831 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927834 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927836 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927839 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927843 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927846 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927848 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.940963 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927851 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927854 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927856 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927859 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927861 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927864 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927866 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927869 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927871 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927875 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927877 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927880 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927883 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927885 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927888 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927891 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927893 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927897 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927901 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.941566 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927905 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927908 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927910 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927913 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927915 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927918 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927920 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927923 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927925 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927928 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927931 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927934 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927937 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927940 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927944 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927947 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927949 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927952 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927955 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927957 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.942023 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927960 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927962 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927965 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927968 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927970 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927973 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927976 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927980 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.927982 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.934565 4988 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.934584 4988 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.934609 4988 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.934614 4988 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934679 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934685 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934689 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934692 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934695 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934698 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934701 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.942603 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934704 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934707 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934710 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934713 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934717 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934719 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934722 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934725 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934728 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934732 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934734 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934737 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934740 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934744 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934749 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934752 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934755 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934758 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934761 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.943212 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934764 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934766 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934769 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934771 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934774 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934777 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934779 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934782 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934785 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934787 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934790 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934792 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934795 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934798 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934800 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934803 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934805 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934808 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934811 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.943785 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934815 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934819 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934822 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934825 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934827 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934830 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934833 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934836 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934839 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934841 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934844 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934847 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934849 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934852 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934854 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934857 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934859 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934862 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934865 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.944330 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934867 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934870 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934873 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934875 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934878 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934881 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934883 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934886 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934888 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934891 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934894 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934896 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934901 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934906 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934910 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934914 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934918 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934922 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934926 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.944870 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934931 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934935 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934938 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934940 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934943 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934946 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934949 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934951 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934954 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934957 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934959 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934962 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934964 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934967 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934969 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934972 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934975 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934977 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934980 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.945328 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934983 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934986 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934988 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.934991 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.935020 4988 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935186 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935190 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935193 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935196 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935198 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935202 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935204 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935207 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935210 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935213 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935215 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.945969 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935218 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935221 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935224 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935226 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935229 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935231 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935234 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935236 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935239 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935242 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935244 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935247 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935249 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935252 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935254 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935256 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935259 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935262 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935265 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.946696 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935268 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935270 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935273 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935275 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935278 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935281 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935284 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935287 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935291 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935293 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935296 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935298 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935301 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935303 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935306 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935309 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935311 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935314 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935317 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.947457 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935320 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935324 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935327 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935330 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935333 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935337 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935340 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935343 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935346 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935349 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935352 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935355 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935357 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935360 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935363 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935365 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935368 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935370 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935373 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.947970 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935376 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935379 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935382 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935384 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935387 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935389 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935392 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935395 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935397 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935400 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935403 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935418 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935426 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935430 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935434 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935439 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935443 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935446 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935449 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935451 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.948551 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935454 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935457 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935459 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935462 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935465 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935467 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935470 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935472 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935475 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935477 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935480 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935483 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935486 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935489 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935491 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935494 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935498 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.949070 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935500 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.935532 4988 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935568 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935572 4988 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935575 4988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935578 4988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935581 4988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935585 4988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935588 4988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935591 4988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935594 4988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935596 4988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935599 4988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935602 4988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935604 4988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935607 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:36:11.949499 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935610 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935613 4988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935616 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935618 4988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935621 4988 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935624 4988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935627 4988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935629 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935632 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935635 4988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935637 4988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935640 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935643 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935645 4988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935648 4988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935651 4988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935654 4988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935657 4988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935660 4988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935662 4988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:36:11.949880 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935665 4988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935667 4988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935670 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935674 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935676 4988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935679 4988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935682 4988 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935684 4988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935686 4988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935689 4988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935692 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935694 4988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935697 4988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935700 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935703 4988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935705 4988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935708 4988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935710 4988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935713 4988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935715 4988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:36:11.950607 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935719 4988 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935721 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935724 4988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935726 4988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935729 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935731 4988 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935734 4988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935737 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935739 4988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935742 4988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935745 4988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935747 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935750 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935752 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935755 4988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935759 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935761 4988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935764 4988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935766 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935769 4988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:36:11.951154 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935774 4988 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935777 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935780 4988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935782 4988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935785 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935787 4988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935790 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935793 4988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935795 4988 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935798 4988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935801 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935803 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935806 4988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935814 4988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935818 4988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935822 4988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935825 4988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935828 4988 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935832 4988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:36:11.951908 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935835 4988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935839 4988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935841 4988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935845 4988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935847 4988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935850 4988 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935853 4988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935855 4988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935859 4988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935862 4988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935864 4988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935867 4988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:11.935871 4988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.935934 4988 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937038 4988 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937099 4988 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937283 4988 server.go:1014] "Starting client certificate rotation" Oct 30 20:36:11.952571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937437 4988 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937481 4988 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:30:29 +0000 UTC" deadline="2028-08-02 23:45:26.394540188 +0000 UTC" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937518 4988 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15411h9m14.457024181s" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.937787 4988 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.938905 4988 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.940514 4988 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.942468 4988 log.go:25] "Validated CRI v1 image API" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.943284 4988 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:36:11.953021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.945460 4988 fs.go:135] Filesystem UUIDs: map[2df800b0-6c15-4695-b98e-453035aabeb1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b6bdb10b-b5f1-481c-a19d-e14546dc99d2:/dev/nvme0n1p3] Oct 30 20:36:11.953305 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.945481 4988 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a3af4a4-5731-41d0-8f14-cbe1543de1f7/volumes/kubernetes.io~projected/kube-api-access-sdngs:{mountpoint:/var/lib/kubelet/pods/0a3af4a4-5731-41d0-8f14-cbe1543de1f7/volumes/kubernetes.io~projected/kube-api-access-sdngs major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a3af4a4-5731-41d0-8f14-cbe1543de1f7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0a3af4a4-5731-41d0-8f14-cbe1543de1f7/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/157b8177-38fc-4986-880f-e820ce5621b1/volumes/kubernetes.io~projected/kube-api-access-sg5jz:{mountpoint:/var/lib/kubelet/pods/157b8177-38fc-4986-880f-e820ce5621b1/volumes/kubernetes.io~projected/kube-api-access-sg5jz major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/157b8177-38fc-4986-880f-e820ce5621b1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/157b8177-38fc-4986-880f-e820ce5621b1/volumes/kubernetes.io~secret/cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~projected/kube-api-access-djkl9:{mountpoint:/var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~projected/kube-api-access-djkl9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9522ce-81d5-40d6-8ac2-ef99be4474c8/volumes/kubernetes.io~projected/kube-api-access-bv6xs:{mountpoint:/var/lib/kubelet/pods/2b9522ce-81d5-40d6-8ac2-ef99be4474c8/volumes/kubernetes.io~projected/kube-api-access-bv6xs major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~projected/kube-api-access-vtvtw:{mountpoint:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~projected/kube-api-access-vtvtw major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d68f309-a728-4000-a631-86ef2c50b3e0/volumes/kubernetes.io~projected/kube-api-access-7hqr4:{mountpoint:/var/lib/kubelet/pods/3d68f309-a728-4000-a631-86ef2c50b3e0/volumes/kubernetes.io~projected/kube-api-access-7hqr4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d68f309-a728-4000-a631-86ef2c50b3e0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3d68f309-a728-4000-a631-86ef2c50b3e0/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~projected/kube-api-access-fcp7p:{mountpoint:/var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~projected/kube-api-access-fcp7p major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/470aba85-18f4-4acc-9e98-6233c2ae3c9c/volumes/kubernetes.io~projected/kube-api-access-4ktzm:{mountpoint:/var/lib/kubelet/pods/470aba85-18f4-4acc-9e98-6233c2ae3c9c/volumes/kubernetes.io~projected/kube-api-access-4ktzm major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa2e4fe-41cb-4690-bba9-206c3aef048a/volumes/kubernetes.io~projected/kube-api-access-rphqg:{mountpoint:/var/lib/kubelet/pods/4fa2e4fe-41cb-4690-bba9-206c3aef048a/volumes/kubernetes.io~projected/kube-api-access-rphqg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5aedd749-52bf-473f-86bb-f42e2cd459d0/volumes/kubernetes.io~projected/kube-api-access-tnxnr:{mountpoint:/var/lib/kubelet/pods/5aedd749-52bf-473f-86bb-f42e2cd459d0/volumes/kubernetes.io~projected/kube-api-access-tnxnr major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b875558-4e2e-4eaf-9dc6-7861f1a95be0/volumes/kubernetes.io~projected/kube-api-access-f28kd:{mountpoint:/var/lib/kubelet/pods/5b875558-4e2e-4eaf-9dc6-7861f1a95be0/volumes/kubernetes.io~projected/kube-api-access-f28kd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f48b68d-7d92-4417-915f-785bb3e457ab/volumes/kubernetes.io~projected/kube-api-access-5xntv:{mountpoint:/var/lib/kubelet/pods/5f48b68d-7d92-4417-915f-785bb3e457ab/volumes/kubernetes.io~projected/kube-api-access-5xntv major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2/volumes/kubernetes.io~projected/kube-api-access-ztvwz:{mountpoint:/var/lib/kubelet/pods/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2/volumes/kubernetes.io~projected/kube-api-access-ztvwz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7092030c-f398-4180-9641-62b31400359a/volumes/kubernetes.io~projected/kube-api-access-cjkvb:{mountpoint:/var/lib/kubelet/pods/7092030c-f398-4180-9641-62b31400359a/volumes/kubernetes.io~projected/kube-api-access-cjkvb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d94d6f-863f-4819-b36f-5d2c753471bd/volumes/kubernetes.io~projected/kube-api-access-9rd8l:{mountpoint:/var/lib/kubelet/pods/70d94d6f-863f-4819-b36f-5d2c753471bd/volumes/kubernetes.io~projected/kube-api-access-9rd8l major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab/volumes/kubernetes.io~projected/kube-api-access-hskcx:{mountpoint:/var/lib/kubelet/pods/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab/volumes/kubernetes.io~projected/kube-api-access-hskcx major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ea62097-316d-49f5-bccf-dba48029bb6d/volumes/kubernetes.io~projected/kube-api-access-pjhrc:{mountpoint:/var/lib/kubelet/pods/7ea62097-316d-49f5-bccf-dba48029bb6d/volumes/kubernetes.io~projected/kube-api-access-pjhrc major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bfb7e64-5a1d-4470-9269-79be9505ebe8/volumes/kubernetes.io~projected/kube-api-access-s2ngt:{mountpoint:/var/lib/kubelet/pods/8bfb7e64-5a1d-4470-9269-79be9505ebe8/volumes/kubernetes.io~projected/kube-api-access-s2ngt major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/975f6dec-89de-4c6e-ab23-a665d5d76ced/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/975f6dec-89de-4c6e-ab23-a665d5d76ced/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99eaa580-8ff6-4a47-971b-80133022ccf0/volumes/kubernetes.io~projected/kube-api-access-xbx6c:{mountpoint:/var/lib/kubelet/pods/99eaa580-8ff6-4a47-971b-80133022ccf0/volumes/kubernetes.io~projected/kube-api-access-xbx6c major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af942c5a-661e-423c-b24b-a7a4a1db3bd9/volumes/kubernetes.io~projected/kube-api-access-zk5qs:{mountpoint:/var/lib/kubelet/pods/af942c5a-661e-423c-b24b-a7a4a1db3bd9/volumes/kubernetes.io~projected/kube-api-access-zk5qs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3970765-0603-41f1-9802-a252c08fdce7/volumes/kubernetes.io~projected/kube-api-access-znd9n:{mountpoint:/var/lib/kubelet/pods/b3970765-0603-41f1-9802-a252c08fdce7/volumes/kubernetes.io~projected/kube-api-access-znd9n major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~projected/kube-api-access-wdn8m:{mountpoint:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~projected/kube-api-access-wdn8m major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c365d189-b04a-4b12-8519-30c074796180/volumes/kubernetes.io~projected/kube-api-access-frqdc:{mountpoint:/var/lib/kubelet/pods/c365d189-b04a-4b12-8519-30c074796180/volumes/kubernetes.io~projected/kube-api-access-frqdc major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c365d189-b04a-4b12-8519-30c074796180/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c365d189-b04a-4b12-8519-30c074796180/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cacd9cad-031b-4a85-a86d-a04eecb18d95/volumes/kubernetes.io~projected/kube-api-access-z5f7r:{mountpoint:/var/lib/kubelet/pods/cacd9cad-031b-4a85-a86d-a04eecb18d95/volumes/kubernetes.io~projected/kube-api-access-z5f7r major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd681674-072e-4dff-a369-9d956c32b960/volumes/kubernetes.io~projected/kube-api-access-l55m2:{mountpoint:/var/lib/kubelet/pods/cd681674-072e-4dff-a369-9d956c32b960/volumes/kubernetes.io~projected/kube-api-access-l55m2 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd681674-072e-4dff-a369-9d956c32b960/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cd681674-072e-4dff-a369-9d956c32b960/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volumes/kubernetes.io~projected/kube-api-access-fkpbk:{mountpoint:/var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volumes/kubernetes.io~projected/kube-api-access-fkpbk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed47a78a-0c03-4f46-b274-5b1d162d1c73/volumes/kubernetes.io~projected/kube-api-access-cvtmj:{mountpoint:/var/lib/kubelet/pods/ed47a78a-0c03-4f46-b274-5b1d162d1c73/volumes/kubernetes.io~projected/kube-api-access-cvtmj major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed47a78a-0c03-4f46-b274-5b1d162d1c73/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ed47a78a-0c03-4f46-b274-5b1d162d1c73/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee569c03-5200-472a-8348-382a2da0327b/volumes/kubernetes.io~projected/kube-api-access-69v9h:{mountpoint:/var/lib/kubelet/pods/ee569c03-5200-472a-8348-382a2da0327b/volumes/kubernetes.io~projected/kube-api-access-69v9h major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f23150de-f251-49ea-a39c-9517da973316/volumes/kubernetes.io~projected/kube-api-access-vhmm7:{mountpoint:/var/lib/kubelet/pods/f23150de-f251-49ea-a39c-9517da973316/volumes/kubernetes.io~projected/kube-api-access-vhmm7 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f24910a3-693b-4290-aec0-7539fe0115d7/volumes/kubernetes.io~projected/kube-api-access-fsfj6:{mountpoint:/var/lib/kubelet/pods/f24910a3-693b-4290-aec0-7539fe0115d7/volumes/kubernetes.io~projected/kube-api-access-fsfj6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f340b19f-a26a-4514-8785-f0714fc17696/volumes/kubernetes.io~projected/kube-api-access-j5qrj:{mountpoint:/var/lib/kubelet/pods/f340b19f-a26a-4514-8785-f0714fc17696/volumes/kubernetes.io~projected/kube-api-access-j5qrj major:0 minor:215 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fe63ddca404fa3a1a966678f770420078444543531c86bc8d140da52ee168d6a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3e07ff3e5d5940da5321f8b07fdf61866d587fea478be4adf0ab4f23fac76329/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5c0503df336eba80d745952153816ed76357d05904792e42e89ecf036cce6b6d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5fccb470fd27056bac9083dd663a3c6bfdf4f700bd4230584aa9d84e26dee35b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0bcbc095f458b10d7bc6296373970200fb67a5693f12336c3c7cd52d54ac6660/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2353bff47763ffd0f26e6fd5a70e0080558fbe92f42ceae43b0b0e4a65c35c32/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c68149b597800fa7952dd9b18f1b90656c2a425d4146deade892c368ca483474/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/966e4b36fef1a53bc73d26d79e68d307ffecdd6dcb15442d56ae30e976263f92/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f2135dca3f5dd9abe5bdc8db633648f44cf1bac94313b3b735dd14182bf458cd/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/f7ee763bb25c07607ee60883c8904fe2e8402ee79207c7b6712887e9d58d9e2a/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/216cd7f1ad3e008ef8bda36d669d46e903fe8648022e90feff370cc5250d7fd0/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/94a1df08e2ed3ae8545de75295165e1e1bf208fbe9321552518b11dafe4a5f57/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/f693ba0326df6027ade88ad3d3714005476e0d8ab4822d44ad951dc2b7bf0f52/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fdd2106c6c26d807d91c2cc2323e2e47749a131abb858d1082aeb46cbf68c63f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/803266936f82586edf6f5c6c76f5df426ae4916fdd09a36ac812d87f61d11557/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8baafe366bcf66fca8d3e1eb010507049eb0393acaa8376a26bdc4f1b14e52c4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9a15050c205419cfdd3e8a5f5dbf6c84cb742804cb36038f3e5304cbde3dee03/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7043b37a7eed2fafcfef55f4e29d56d8a006a40f58a040df2d946bfa95a6913e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2acf397279762ceb4ef6d79ce3741ffa250927859baa525fcd9d285f42e4147f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3540bafc45372b5a494d55abfc2a25ce3a673b7ce42b35810416f6510f9d529c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/45db377561723f56e87e9efa2da0051ac6b02f26ffb381da6667531815de7410/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5c577a7b3104e82362969586ef6f112ad4a458188d947c2971ad03c8d917235e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7dbcb3a484eaf3d5af2ecd523cf1266283fb3084988ce241c707e787f6ca8bad/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f6aff780b1006f8e09776d8a62c6816ba04f107f665cd06d7a33f5c2771442bf/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/dded9dd4ff319be069b532a949e5372045341e846166e73d3b3af74239b9ae04/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/6a2edbdab1b6658bf633a0b29327e394a38d0be9d894593c574d54b667d233e9/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/4795bab2af108252be6a52366e7c8b94be764a142fa092b47f95245f809cdfbe/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/97c0c3f4dd39fbc31f12305d35528343eb4d2b56822ade22f6fd67ed6067ad34/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/a40c6a4b47860e70fde42ab19e89c4c15437ef8fdf37ed10ae43706edd9d28fe/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/372be9efed48026fc0d8842f786786cfb441f462898f9f22f55858210f9280df/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/55b3c64fffd01385e29dbd649455cb0af7aaf805d0086e80afd4efd6bd2e322c/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/2fd434486a68f59bb71b2eb600efcebb8904e6d3b6c0c3369396f5349cf9bb25/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/48f19d24eb7a0a058d42008c4199227babc2ba400109ff96061f8fef4ce976ef/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/36bdea38ebfb3b08aba450c6891adc01d945a4d0bd5478b01fcbfc53448d15d5/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/d7b2ce59c3e7d101c13016bb14960534028058e9961b8bf65729b7096df02851/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/353f30773ae84d99b8b207f82d310dfbcd2267b5cd469cf3357801ede90d64b4/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/cdfc690a8d0ea88832bac18f02cc3d0289fdef0b9fecfd578b20085e75dd1f70/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/4011e43e37216ed13f2b15d416962e94eed167f9a314e9c4b06ebab78d57a9dd/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/846ceb3715d4b696f33f52b809c5e12196ff2c86b363777d9cc25e649f62f4f5/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6c00e9ea02674615eb66c4590ad277c6391da62461d09cd0e942dc7ad3f9ede5/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/04a4f69b9762a81294cc5a004c5909002651ca531ab4c98aaa738939fe6a78cd/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/fb9fd6bb2f6c4b8c8863eebdb5059663aea9a507e04910f6f2d2351eeb83c2f6/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/860b541b198e190053d367e56e921d96d153162e14c42db9d0f25ae548972563/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/9371444dc5ef6e7e1b8b0bfefd23631e2b93e925dd97fee54bf30c6a9efb538a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/7705289bdfdf9d7dae81a2d09dcaefe5e755ad4d2beafc821db72fa43702e6a7/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/0e4a3bac45a8975102970a7a1d0742da340dbc62fc0bfe9fa46d4b31c9efa8ce/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/814e68bb65ad62d5d1be5303277c08f6abe3f51c2028dcf4db568e53e565eec3/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/83b5cfc17d8775ef8ec820dd20ad9cb956c86f1461d0b0ef77f80bb4ec1b6a50/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/1d864a61a9cf973082f80e98a393c8ef53a350110b64fc14b93ba35ddd7c96dc/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/416af71cce8a8aa3b2ff0cb5521c74cfb0bc97ef4bc622cb5451ab5bbd6e6d16/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/c02d2d8637b0a4ef3fb22397720427591c2f2999c2620fccc7eaa387ad18866d/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/ed840af64aeed55799b7685b6798cebe924f762398831bd9ca2549afc6c7bf5e/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/db211d3b02d778a9c6892357ac548d146e01494f0aac50e1e1bc60aba9942739/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ea720b2ff2304a738f35278abe76286e7fce20d4ca9dbd2dfade31034ebe78c3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/975febe2165c15e0b45616e7fc4b8b5d7ebd94f1b33d50ff5914a86dd3a63120/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f7be9f15afd515ac1c642808d81d27231165d63c3f4553b8b2f831d6c6b87b9b/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/84598939b4057eb052cc34bd81f53772ef22ae15eb07f38d36e55214cec5c00f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/fa749f7cc78cfc49cab9e271e2a81473ccde44bc54b9b98818982804fdd1c8b0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ddcddcbd39cae34cb753970366ba951d997105478d420d6355230008b65d3995/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d82101acbf315944528b2be2bafe22a6ced4d0215c3a2cd903dd95fee31e7bfb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/83f2ff775b8d313573d9691dc8911fb2edae5a2ca78130b6ed6eac45fd1901da/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/566e6cba2659cbf68d1336785ff0ee0610e7c2cc7f948dcd0616e98d6ae69e38/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 20:36:11.956748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.956311 4988 manager.go:217] Machine: {Timestamp:2026-10-30 20:36:11.955809527 +0000 UTC m=+0.063305431 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097108 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc278e493b110fdc3f776d45ff989 SystemUUID:ec2dc278-e493-b110-fdc3-f776d45ff989 BootID:f0e261aa-311f-48a3-9cf7-d362ea654d48 Filesystems:[{Device:/var/lib/kubelet/pods/470aba85-18f4-4acc-9e98-6233c2ae3c9c/volumes/kubernetes.io~projected/kube-api-access-4ktzm DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab/volumes/kubernetes.io~projected/kube-api-access-hskcx DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59597831c0d8e45730fe0b19c58d08105355af84bad5cc49ed909fb2c21cc3f6/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f59d7314403e6e2a56fabd9c31d610f365a32f5d9d589b961c05ba8e8769bcca/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~projected/kube-api-access-fcp7p DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37bb7b892f8285924b4bffee2691cc15ce4ffd4f3be342130ba77fc721592c96/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f48b68d-7d92-4417-915f-785bb3e457ab/volumes/kubernetes.io~projected/kube-api-access-5xntv DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee569c03-5200-472a-8348-382a2da0327b/volumes/kubernetes.io~projected/kube-api-access-69v9h DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f12459757d55309e34e9c9ee46c95c6274e02e14b576003e85613a51dccf787/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62f00eafbd2875b6aa9d979a3fdb8b410dc751a1767ea01f1d34a751a40b1c9b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3970765-0603-41f1-9802-a252c08fdce7/volumes/kubernetes.io~projected/kube-api-access-znd9n DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/af942c5a-661e-423c-b24b-a7a4a1db3bd9/volumes/kubernetes.io~projected/kube-api-access-zk5qs DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70d94d6f-863f-4819-b36f-5d2c753471bd/volumes/kubernetes.io~projected/kube-api-access-9rd8l DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c365d189-b04a-4b12-8519-30c074796180/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f340b19f-a26a-4514-8785-f0714fc17696/volumes/kubernetes.io~projected/kube-api-access-j5qrj DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55eabdeffe6425715cc798534fee185e5aa92e6934d5bf7704a3e84dcb528e1d/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/975f6dec-89de-4c6e-ab23-a665d5d76ced/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8089437d7bfba95ec87625185a0437d463cfe8bffda751bf10121b2100478c88/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~projected/kube-api-access-vtvtw DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/433d289f554b9157232c2ec54f56471370618032957b08df892845097ad248d7/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed47a78a-0c03-4f46-b274-5b1d162d1c73/volumes/kubernetes.io~projected/kube-api-access-cvtmj DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f23150de-f251-49ea-a39c-9517da973316/volumes/kubernetes.io~projected/kube-api-access-vhmm7 DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f24910a3-693b-4290-aec0-7539fe0115d7/volumes/kubernetes.io~projected/kube-api-access-fsfj6 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/740a13b30a3c61d8dbc114993318d0ef86a1a187facf70c70ecbd4c522bd942a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d4b1ec4caa45ec8ae068c4f15a3901ea77eee5365f18a88e6e90d02a579643d/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff27d82261f36ed05ef784c65272757197bdb47f6694fd335cb6d1040a76df5/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ea62097-316d-49f5-bccf-dba48029bb6d/volumes/kubernetes.io~projected/kube-api-access-pjhrc DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e8b4ccc9aac42f15d59429618795a67919280ad6f309a8efeefe16020d3bb08/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cacd9cad-031b-4a85-a86d-a04eecb18d95/volumes/kubernetes.io~projected/kube-api-access-z5f7r DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/157b8177-38fc-4986-880f-e820ce5621b1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bfb7e64-5a1d-4470-9269-79be9505ebe8/volumes/kubernetes.io~projected/kube-api-access-s2ngt DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e933dad7-264e-49fc-ae45-e6ba8ffad936/volumes/kubernetes.io~projected/kube-api-access-fkpbk DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~projected/kube-api-access-djkl9 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b875558-4e2e-4eaf-9dc6-7861f1a95be0/volumes/kubernetes.io~projected/kube-api-access-f28kd DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f42849fcec439486e3e05df6b36b261df16bbc94aa561feaed0645b96985d6d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66ce946701dc07696d763b975709833b89ecb9cd6cb3270996cc703b96afa01d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d68f309-a728-4000-a631-86ef2c50b3e0/volumes/kubernetes.io~projected/kube-api-access-7hqr4 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dda72cb748f9f676e8e8a90ba5e7aa267cace1c78c3ad4ae5f6b7ec115000acc/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c365d189-b04a-4b12-8519-30c074796180/volumes/kubernetes.io~projected/kube-api-access-frqdc DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5aedd749-52bf-473f-86bb-f42e2cd459d0/volumes/kubernetes.io~projected/kube-api-access-tnxnr DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41b7e2e0ce4929f56328262377c0abe79b5e8dc4a0a1e563da3bb5341c1f5d5b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c0f4ca2e33af4775678d551cb7687079945f0e7604ce2c410e6eec5742c6b9b/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed47a78a-0c03-4f46-b274-5b1d162d1c73/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2/volumes/kubernetes.io~projected/kube-api-access-ztvwz DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ea47fc6-4d87-4c4f-b623-801cfe65c047/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01827abc78c24593712f91aa7292ab9a15b0750acbb436c09182620f2a78be56/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9522ce-81d5-40d6-8ac2-ef99be4474c8/volumes/kubernetes.io~projected/kube-api-access-bv6xs DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~projected/kube-api-access-wdn8m DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd681674-072e-4dff-a369-9d956c32b960/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a3af4a4-5731-41d0-8f14-cbe1543de1f7/volumes/kubernetes.io~projected/kube-api-access-sdngs DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77ed0961bb6d897d73ac6106774a723851be1ec78b8214dec4d545c96f590968/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7092030c-f398-4180-9641-62b31400359a/volumes/kubernetes.io~projected/kube-api-access-cjkvb DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3efc8cc8-17bc-426c-84e1-43e4d7888e1c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e785fe8bc0ec4af5c7e82273d42fc172bb4d390c72dbe954cbf9c77f1f3e2f4/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa2e4fe-41cb-4690-bba9-206c3aef048a/volumes/kubernetes.io~projected/kube-api-access-rphqg DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/939828b9535b599d02d5d23b58adf860614835d170e1df19e20414cf0e34570a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/157b8177-38fc-4986-880f-e820ce5621b1/volumes/kubernetes.io~projected/kube-api-access-sg5jz DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd681674-072e-4dff-a369-9d956c32b960/volumes/kubernetes.io~projected/kube-api-access-l55m2 DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b451dde6f732612b39e3dcb70efdc7586b13e62bc0828d37dfa560064886878/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7514173e3106f0478afaff689f899919a50b53ac67cff4a5463bd63d34910fbd/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a3af4a4-5731-41d0-8f14-cbe1543de1f7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d68f309-a728-4000-a631-86ef2c50b3e0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99eaa580-8ff6-4a47-971b-80133022ccf0/volumes/kubernetes.io~projected/kube-api-access-xbx6c DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b451dde6f73261 MacAddress:0e:50:37:25:e0:4c Speed:10000 Mtu:8901} {Name:37bb7b892f82859 MacAddress:46:f7:6e:49:99:44 Speed:10000 Mtu:8901} {Name:433d289f554b915 MacAddress:7a:63:a5:91:2b:0d Speed:10000 Mtu:8901} {Name:55eabdeffe64257 MacAddress:8a:64:81:18:a6:f9 Speed:10000 Mtu:8901} {Name:59597831c0d8e45 MacAddress:5e:50:f6:30:73:64 Speed:10000 Mtu:8901} {Name:5f12459757d5530 MacAddress:8e:56:e7:d6:a9:bf Speed:10000 Mtu:8901} {Name:6142043c46f4f68 MacAddress:ea:a4:d8:a1:de:6b Speed:10000 Mtu:8901} {Name:62f00eafbd2875b MacAddress:7e:e6:7d:c4:a1:f4 Speed:10000 Mtu:8901} {Name:7514173e3106f04 MacAddress:ee:72:42:1b:cf:cd Speed:10000 Mtu:8901} {Name:77ed0961bb6d897 MacAddress:e2:88:e8:b9:e0:dc Speed:10000 Mtu:8901} {Name:7c0f4ca2e33af47 MacAddress:56:fa:60:ab:46:ed Speed:10000 Mtu:8901} {Name:7e8b4ccc9aac42f MacAddress:1a:1c:ae:e5:66:01 Speed:10000 Mtu:8901} {Name:7f42849fcec4394 MacAddress:3e:8a:02:4a:fe:5a Speed:10000 Mtu:8901} {Name:8089437d7bfba95 MacAddress:66:04:1a:0f:4c:e5 Speed:10000 Mtu:8901} {Name:8d4b1ec4caa45ec MacAddress:f6:cd:a0:bc:ba:04 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:55:df:02:73:df Speed:0 Mtu:9001} {Name:br-int MacAddress:82:47:c3:e6:b9:13 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:55:df:02:73:df Speed:0 Mtu:9001} {Name:f59d7314403e6e2 MacAddress:6e:d6:c7:db:e5:67 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6a:e0:ef:88:b4:3b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:79:f9:b1:85:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:36:11.956748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.956752 4988 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:36:11.957018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.956840 4988 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:36:11.957151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957129 4988 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:36:11.957288 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957151 4988 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-115.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:36:11.957332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957291 4988 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:36:11.957332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957299 4988 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:36:11.957332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957311 4988 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:36:11.957332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957331 4988 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:36:11.957523 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957442 4988 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:36:11.957624 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957612 4988 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:36:11.957755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957744 4988 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:36:11.957833 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957823 4988 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:36:11.957862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957844 4988 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:36:11.957862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957858 4988 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:36:11.957920 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957872 4988 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:36:11.957920 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.957883 4988 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:36:11.958856 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.958839 4988 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:36:11.958856 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.958855 4988 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:36:11.959835 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.959816 4988 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:36:11.961966 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.961952 4988 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:36:11.962010 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.961966 4988 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:36:11.962068 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962059 4988 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 20:36:11.962252 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962242 4988 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:36:11.962282 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962271 4988 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:36:11.962367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962359 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962373 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962380 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962385 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962390 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962397 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962403 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962426 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962433 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962438 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:36:11.962403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962447 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:36:11.962775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962457 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:36:11.962775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962485 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:36:11.962775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.962491 4988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:36:11.965947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.965934 4988 server.go:1263] "Started kubelet" Oct 30 20:36:11.966046 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.965996 4988 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:36:11.966297 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.966112 4988 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:36:11.966297 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.966260 4988 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:36:11.966824 ip-10-0-141-115 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:36:11.967129 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.967048 4988 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:36:11.969019 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.968721 4988 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973708 4988 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973820 4988 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973834 4988 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:11.973850 4988 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973715 4988 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973935 4988 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973922 4988 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:30:31 +0000 UTC" deadline="2028-05-15 04:56:36.600284203 +0000 UTC" Oct 30 20:36:11.974518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.973958 4988 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13496h20m24.626329488s" Oct 30 20:36:11.975115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.974715 4988 factory.go:55] Registering systemd factory Oct 30 20:36:11.975115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.974732 4988 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:36:11.975548 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.975518 4988 factory.go:153] Registering CRI-O factory Oct 30 20:36:11.975645 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.975553 4988 factory.go:223] Registration of the crio container factory successfully Oct 30 20:36:11.975645 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.975613 4988 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:36:11.975645 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.975635 4988 factory.go:103] Registering Raw factory Oct 30 20:36:11.975792 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.975650 4988 manager.go:1199] Started watching for new ooms in manager Oct 30 20:36:11.975973 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.975961 4988 manager.go:319] Starting recovery of all containers Oct 30 20:36:11.977059 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977017 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977059 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7092030c-f398-4180-9641-62b31400359a" volumeName="kubernetes.io/projected/7092030c-f398-4180-9641-62b31400359a-kube-api-access-cjkvb" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977075 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99eaa580-8ff6-4a47-971b-80133022ccf0" volumeName="kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977088 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e933dad7-264e-49fc-ae45-e6ba8ffad936" volumeName="kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-config" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977102 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e933dad7-264e-49fc-ae45-e6ba8ffad936" volumeName="kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-script-lib" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977115 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="157b8177-38fc-4986-880f-e820ce5621b1" volumeName="kubernetes.io/projected/157b8177-38fc-4986-880f-e820ce5621b1-kube-api-access-sg5jz" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977127 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977139 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efc8cc8-17bc-426c-84e1-43e4d7888e1c" volumeName="kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-default-certificate" seLinuxMountContext="" Oct 30 20:36:11.977157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977153 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="975f6dec-89de-4c6e-ab23-a665d5d76ced" volumeName="kubernetes.io/configmap/975f6dec-89de-4c6e-ab23-a665d5d76ced-konnectivity-ca" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977166 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3970765-0603-41f1-9802-a252c08fdce7" volumeName="kubernetes.io/projected/b3970765-0603-41f1-9802-a252c08fdce7-kube-api-access-znd9n" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977178 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c365d189-b04a-4b12-8519-30c074796180" volumeName="kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-service-ca-bundle" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977190 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e933dad7-264e-49fc-ae45-e6ba8ffad936" volumeName="kubernetes.io/secret/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977203 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f24910a3-693b-4290-aec0-7539fe0115d7" volumeName="kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-cni-binary-copy" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977218 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a3af4a4-5731-41d0-8f14-cbe1543de1f7" volumeName="kubernetes.io/projected/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-kube-api-access-sdngs" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977232 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9522ce-81d5-40d6-8ac2-ef99be4474c8" volumeName="kubernetes.io/projected/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-kube-api-access-bv6xs" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977256 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977268 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aedd749-52bf-473f-86bb-f42e2cd459d0" volumeName="kubernetes.io/projected/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-api-access-tnxnr" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977281 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99eaa580-8ff6-4a47-971b-80133022ccf0" volumeName="kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977293 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99eaa580-8ff6-4a47-971b-80133022ccf0" volumeName="kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977303 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af942c5a-661e-423c-b24b-a7a4a1db3bd9" volumeName="kubernetes.io/configmap/af942c5a-661e-423c-b24b-a7a4a1db3bd9-iptables-alerter-script" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977314 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af942c5a-661e-423c-b24b-a7a4a1db3bd9" volumeName="kubernetes.io/projected/af942c5a-661e-423c-b24b-a7a4a1db3bd9-kube-api-access-zk5qs" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977325 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d68f309-a728-4000-a631-86ef2c50b3e0" volumeName="kubernetes.io/projected/3d68f309-a728-4000-a631-86ef2c50b3e0-kube-api-access-7hqr4" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977336 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ed71fea-5f6c-4ba4-be09-b524c1abfc4a" volumeName="kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977347 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b81cd8d-9d74-4e6d-bda5-7b26aa16feab" volumeName="kubernetes.io/projected/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab-kube-api-access-hskcx" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977357 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ea62097-316d-49f5-bccf-dba48029bb6d" volumeName="kubernetes.io/projected/7ea62097-316d-49f5-bccf-dba48029bb6d-kube-api-access-pjhrc" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977369 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb7e64-5a1d-4470-9269-79be9505ebe8" volumeName="kubernetes.io/empty-dir/8bfb7e64-5a1d-4470-9269-79be9505ebe8-tmp-dir" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977381 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99eaa580-8ff6-4a47-971b-80133022ccf0" volumeName="kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977404 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="157b8177-38fc-4986-880f-e820ce5621b1" volumeName="kubernetes.io/secret/157b8177-38fc-4986-880f-e820ce5621b1-cert" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977438 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa2e4fe-41cb-4690-bba9-206c3aef048a" volumeName="kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977449 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa2e4fe-41cb-4690-bba9-206c3aef048a" volumeName="kubernetes.io/projected/4fa2e4fe-41cb-4690-bba9-206c3aef048a-kube-api-access-rphqg" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977462 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977488 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db8922f9-31fe-4afd-b227-313eecf0fcaa" volumeName="kubernetes.io/configmap/db8922f9-31fe-4afd-b227-313eecf0fcaa-nginx-conf" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977499 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f23150de-f251-49ea-a39c-9517da973316" volumeName="kubernetes.io/projected/f23150de-f251-49ea-a39c-9517da973316-kube-api-access-vhmm7" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977512 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f24910a3-693b-4290-aec0-7539fe0115d7" volumeName="kubernetes.io/projected/f24910a3-693b-4290-aec0-7539fe0115d7-kube-api-access-fsfj6" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977525 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa2e4fe-41cb-4690-bba9-206c3aef048a" volumeName="kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977536 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b875558-4e2e-4eaf-9dc6-7861f1a95be0" volumeName="kubernetes.io/empty-dir/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-tmp-dir" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977548 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration" seLinuxMountContext="" Oct 30 20:36:11.977551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977560 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c365d189-b04a-4b12-8519-30c074796180" volumeName="kubernetes.io/projected/c365d189-b04a-4b12-8519-30c074796180-kube-api-access-frqdc" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977572 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d68f309-a728-4000-a631-86ef2c50b3e0" volumeName="kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-trusted-ca" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977583 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f48b68d-7d92-4417-915f-785bb3e457ab" volumeName="kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977596 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977607 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd681674-072e-4dff-a369-9d956c32b960" volumeName="kubernetes.io/secret/cd681674-072e-4dff-a369-9d956c32b960-serving-cert" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977620 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee569c03-5200-472a-8348-382a2da0327b" volumeName="kubernetes.io/projected/ee569c03-5200-472a-8348-382a2da0327b-kube-api-access-69v9h" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977632 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f340b19f-a26a-4514-8785-f0714fc17696" volumeName="kubernetes.io/projected/f340b19f-a26a-4514-8785-f0714fc17696-kube-api-access-j5qrj" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977644 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a3af4a4-5731-41d0-8f14-cbe1543de1f7" volumeName="kubernetes.io/configmap/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-config" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977658 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d94d6f-863f-4819-b36f-5d2c753471bd" volumeName="kubernetes.io/projected/70d94d6f-863f-4819-b36f-5d2c753471bd-kube-api-access-9rd8l" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977671 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e933dad7-264e-49fc-ae45-e6ba8ffad936" volumeName="kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-env-overrides" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977682 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ea62097-316d-49f5-bccf-dba48029bb6d" volumeName="kubernetes.io/configmap/7ea62097-316d-49f5-bccf-dba48029bb6d-telemetry-config" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977695 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977707 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c365d189-b04a-4b12-8519-30c074796180" volumeName="kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-snapshots" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977720 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ea47fc6-4d87-4c4f-b623-801cfe65c047" volumeName="kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-tuned" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977733 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9522ce-81d5-40d6-8ac2-ef99be4474c8" volumeName="kubernetes.io/configmap/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-config" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977748 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977762 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efc8cc8-17bc-426c-84e1-43e4d7888e1c" volumeName="kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-stats-auth" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977775 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977788 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c365d189-b04a-4b12-8519-30c074796180" volumeName="kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-trusted-ca-bundle" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977809 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd9cad-031b-4a85-a86d-a04eecb18d95" volumeName="kubernetes.io/projected/cacd9cad-031b-4a85-a86d-a04eecb18d95-kube-api-access-z5f7r" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977821 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed47a78a-0c03-4f46-b274-5b1d162d1c73" volumeName="kubernetes.io/configmap/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-cabundle" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977835 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb7e64-5a1d-4470-9269-79be9505ebe8" volumeName="kubernetes.io/configmap/8bfb7e64-5a1d-4470-9269-79be9505ebe8-config-volume" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977849 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd681674-072e-4dff-a369-9d956c32b960" volumeName="kubernetes.io/configmap/cd681674-072e-4dff-a369-9d956c32b960-config" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977862 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e933dad7-264e-49fc-ae45-e6ba8ffad936" volumeName="kubernetes.io/projected/e933dad7-264e-49fc-ae45-e6ba8ffad936-kube-api-access-fkpbk" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977874 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed47a78a-0c03-4f46-b274-5b1d162d1c73" volumeName="kubernetes.io/projected/ed47a78a-0c03-4f46-b274-5b1d162d1c73-kube-api-access-cvtmj" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977886 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bfb7e64-5a1d-4470-9269-79be9505ebe8" volumeName="kubernetes.io/projected/8bfb7e64-5a1d-4470-9269-79be9505ebe8-kube-api-access-s2ngt" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977900 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977913 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a3af4a4-5731-41d0-8f14-cbe1543de1f7" volumeName="kubernetes.io/secret/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-serving-cert" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977926 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ea47fc6-4d87-4c4f-b623-801cfe65c047" volumeName="kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-tmp" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977937 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977949 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977961 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d68f309-a728-4000-a631-86ef2c50b3e0" volumeName="kubernetes.io/secret/3d68f309-a728-4000-a631-86ef2c50b3e0-serving-cert" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977972 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa2e4fe-41cb-4690-bba9-206c3aef048a" volumeName="kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-binary-copy" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.977987 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="975f6dec-89de-4c6e-ab23-a665d5d76ced" volumeName="kubernetes.io/secret/975f6dec-89de-4c6e-ab23-a665d5d76ced-agent-certs" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978000 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b875558-4e2e-4eaf-9dc6-7861f1a95be0" volumeName="kubernetes.io/projected/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-kube-api-access-f28kd" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978014 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2" volumeName="kubernetes.io/projected/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kube-api-access-ztvwz" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978026 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c365d189-b04a-4b12-8519-30c074796180" volumeName="kubernetes.io/secret/c365d189-b04a-4b12-8519-30c074796180-serving-cert" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978039 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd681674-072e-4dff-a369-9d956c32b960" volumeName="kubernetes.io/projected/cd681674-072e-4dff-a369-9d956c32b960-kube-api-access-l55m2" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978052 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f24910a3-693b-4290-aec0-7539fe0115d7" volumeName="kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-multus-daemon-config" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978067 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ea47fc6-4d87-4c4f-b623-801cfe65c047" volumeName="kubernetes.io/projected/1ea47fc6-4d87-4c4f-b623-801cfe65c047-kube-api-access-djkl9" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978080 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7092030c-f398-4180-9641-62b31400359a" volumeName="kubernetes.io/configmap/7092030c-f398-4180-9641-62b31400359a-serviceca" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978095 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31e37c-2894-420c-84cc-16fa0776f92c" volumeName="kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978108 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed47a78a-0c03-4f46-b274-5b1d162d1c73" volumeName="kubernetes.io/secret/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-key" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978121 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391e9838-62c0-4d2d-8d8e-29d7493eff20" volumeName="kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978139 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d68f309-a728-4000-a631-86ef2c50b3e0" volumeName="kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-config" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978154 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efc8cc8-17bc-426c-84e1-43e4d7888e1c" volumeName="kubernetes.io/projected/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-kube-api-access-fcp7p" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978166 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="470aba85-18f4-4acc-9e98-6233c2ae3c9c" volumeName="kubernetes.io/projected/470aba85-18f4-4acc-9e98-6233c2ae3c9c-kube-api-access-4ktzm" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978179 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aedd749-52bf-473f-86bb-f42e2cd459d0" volumeName="kubernetes.io/empty-dir/5aedd749-52bf-473f-86bb-f42e2cd459d0-data-volume" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978192 4988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c365d189-b04a-4b12-8519-30c074796180" volumeName="kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-tmp" seLinuxMountContext="" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978205 4988 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:36:11.978945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.978216 4988 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:36:11.996554 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.996527 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:36:11.996695 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:11.996606 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:36:12.000340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.000307 4988 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:36:12.009237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.009192 4988 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:36:12.009237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.009238 4988 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:36:12.009692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.009260 4988 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:36:12.009692 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.009306 4988 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 20:36:12.018920 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.018902 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:36:12.019014 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.018925 4988 generic.go:356] "Generic (PLEG): container finished" podID="3d68f309-a728-4000-a631-86ef2c50b3e0" containerID="48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557" exitCode=255 Oct 30 20:36:12.028243 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.028214 4988 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="fed5b5e052d9bc06540065aa2143c7df574d1f3921bd81bd6050c0468ef111f4" exitCode=0 Oct 30 20:36:12.028243 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.028244 4988 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="b1e6e74fdd6df69ee02a8d27e860a9bc88f5c58aebccf5c6c61eca5410eb1b57" exitCode=0 Oct 30 20:36:12.028243 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.028254 4988 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="8d1c778a797b447374e36c5b70273ac0b0c2e4ecc1a0ac5b8c755f19ccfb48fe" exitCode=0 Oct 30 20:36:12.028485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.028262 4988 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="6dff6f32b0d90f2aa7b50f9458cf1c84eacdecadf022326bfb2b331da4ac83a1" exitCode=0 Oct 30 20:36:12.028485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.028269 4988 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="0e5a01bd4629c7bf118782d8314396d2f3680a1a7d323b4bdae9026a7ea66815" exitCode=0 Oct 30 20:36:12.028485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.028277 4988 generic.go:356] "Generic (PLEG): container finished" podID="4fa2e4fe-41cb-4690-bba9-206c3aef048a" containerID="6960e9e81a926f3f80627850921f673fe15477ca8b93f9fe56d61a60526a65e4" exitCode=0 Oct 30 20:36:12.051926 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.051899 4988 generic.go:356] "Generic (PLEG): container finished" podID="28735f4653fbe0f3b3d64a88696fbee4" containerID="2a1463e9b6c511d9e225c2035bf7c47a7d86627c94ff5c378053ac7952694382" exitCode=0 Oct 30 20:36:12.073924 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.073893 4988 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:36:12.083928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.083907 4988 manager.go:324] Recovery completed Oct 30 20:36:12.105381 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.105306 4988 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:36:12.106772 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.106747 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:36:12.106870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.106786 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:36:12.106870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.106798 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:36:12.109227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109210 4988 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:36:12.109227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109223 4988 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:36:12.109338 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109224 4988 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:36:12.109338 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109298 4988 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:36:12.109430 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.109360 4988 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 20:36:12.109480 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109466 4988 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 20:36:12.109509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109481 4988 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 20:36:12.109509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109498 4988 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 20:36:12.109509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109504 4988 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 20:36:12.109509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109509 4988 policy_none.go:50] "Start" Oct 30 20:36:12.109622 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109523 4988 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:36:12.109622 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.109532 4988 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:36:12.110265 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.110249 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:36:12.110333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.110276 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:36:12.110333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.110286 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:36:12.121312 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.121295 4988 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 20:36:12.121386 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.121317 4988 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 20:36:12.121386 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.121324 4988 policy_none.go:44] "Start" Oct 30 20:36:12.125835 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.125807 4988 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:36:12.128397 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.128382 4988 manager.go:341] "Starting Device Plugin manager" Oct 30 20:36:12.128494 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.128426 4988 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:36:12.128494 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.128440 4988 server.go:87] "Starting device plugin registration server" Oct 30 20:36:12.128759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.128747 4988 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:36:12.128796 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.128761 4988 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:36:12.128875 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.128854 4988 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:36:12.129003 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.128961 4988 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:36:12.129116 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.129009 4988 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:36:12.129116 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.129017 4988 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:36:12.129903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.129072 4988 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:36:12.12899397Z","UUID":"f302bd00-0d81-469c-a7e8-6e6245190e2a","Handler":null,"Name":"","Endpoint":""} Oct 30 20:36:12.130598 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.130531 4988 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:36:12.130598 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.130577 4988 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-115.us-west-1.compute.internal\" not found" Oct 30 20:36:12.131149 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.131127 4988 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:36:12.131200 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.131153 4988 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:36:12.171761 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.171693 4988 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:36:12.171761 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.171753 4988 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:36:12.171964 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.171859 4988 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 20:36:12.171964 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.171909 4988 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:36:12.221490 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.221450 4988 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:36:12.229484 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.229462 4988 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:36:12.230653 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.230636 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:36:12.230738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.230671 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:36:12.230738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.230682 4988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-115.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 20:36:12.230738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.230731 4988 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.309542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.309473 4988 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal"] Oct 30 20:36:12.310440 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310401 4988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.310626 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310603 4988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.310751 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310500 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"28735f4653fbe0f3b3d64a88696fbee4","Type":"ContainerStarted","Data":"8d333249fb913757e581e3efd2a96c167a273b593b68b7302d9b4f33d80ebf18"} Oct 30 20:36:12.310838 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310803 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"28735f4653fbe0f3b3d64a88696fbee4","Type":"ContainerDied","Data":"2a1463e9b6c511d9e225c2035bf7c47a7d86627c94ff5c378053ac7952694382"} Oct 30 20:36:12.311099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310866 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"28735f4653fbe0f3b3d64a88696fbee4","Type":"ContainerStarted","Data":"1f64690e2b6c425be4eace8d3e0edde9c288d18ec668c26f0db7d37160605c39"} Oct 30 20:36:12.311099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310889 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"124b8400391b2921b52ea70b658a260b","Type":"ContainerStarted","Data":"a1ccb0ddf6524571724be344192204fc913e630f44075c6e7a4fc60f7194d215"} Oct 30 20:36:12.311099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.310908 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" event={"ID":"124b8400391b2921b52ea70b658a260b","Type":"ContainerStarted","Data":"4fcc4784e39b4db30056e6dd7a30af62a4ca1fab2ed0d69208e9b5b6ebf59590"} Oct 30 20:36:12.339838 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.339809 4988 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.339967 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.339888 4988 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.373765 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.373697 4988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:36:12.373765 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.373752 4988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.374269 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.374247 4988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:36:12.374434 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:12.374287 4988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.473904 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.473864 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/124b8400391b2921b52ea70b658a260b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"124b8400391b2921b52ea70b658a260b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.474083 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.473926 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.474083 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.473951 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.574790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.574740 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/124b8400391b2921b52ea70b658a260b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"124b8400391b2921b52ea70b658a260b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.574987 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.574802 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.574987 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.574838 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.574987 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.574850 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/124b8400391b2921b52ea70b658a260b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"124b8400391b2921b52ea70b658a260b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.574987 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.574901 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.574987 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.574932 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28735f4653fbe0f3b3d64a88696fbee4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" (UID: \"28735f4653fbe0f3b3d64a88696fbee4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:12.958626 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:12.958588 4988 apiserver.go:52] "Watching apiserver" Oct 30 20:36:13.071001 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.070964 4988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:13.071170 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.071154 4988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:13.171447 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.171402 4988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:36:13.171613 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.171464 4988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:13.171613 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.171403 4988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:36:13.171613 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.171605 4988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:36:13.172275 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.172260 4988 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:36:13.173156 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.173131 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-l2csm","openshift-image-registry/image-registry-587bd9b797-w6c54","openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf","openshift-insights/insights-runtime-extractor-bdlwf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2","openshift-dns/node-resolver-7kw6v","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv","openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n","openshift-ingress-canary/ingress-canary-s6pl6","openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h","kube-system/kas-connection-checker-685895c7c9-tkhfn","openshift-image-registry/image-registry-85df8d78bd-v8dx4","kube-system/global-pull-secret-syncer-2kjtc","openshift-ingress/router-default-789f8dddc5-m74zk","openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx","kube-system/kas-connection-checker-685895c7c9-ldcc4","openshift-insights/insights-operator-7c4986d449-tmxhb","openshift-image-registry/node-ca-7dvbw","openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp","openshift-ovn-kubernetes/ovnkube-node-27nvb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c","openshift-multus/multus-r7m68","openshift-insights/periodic-gathering-jfr7m-mw5r6","openshift-network-console/networking-console-plugin-86c478bb88-zbpwc","openshift-cluster-node-tuning-operator/tuned-wfcwm","openshift-service-ca/service-ca-5fffc97bdf-h4ncf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz","openshift-network-diagnostics/network-check-target-jdq2q","kube-system/kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-7bt9s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-115.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-8ngvj","kube-system/kas-connection-checker-685895c7c9-sg6jp","openshift-multus/multus-additional-cni-plugins-4fv75","openshift-console-operator/console-operator-7cd56b8dbc-2lsbc","openshift-dns/dns-default-8jh5r"] Oct 30 20:36:13.175137 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.175117 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.179159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.179134 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktzm\" (UniqueName: \"kubernetes.io/projected/470aba85-18f4-4acc-9e98-6233c2ae3c9c-kube-api-access-4ktzm\") pod \"kas-connection-checker-685895c7c9-sg6jp\" (UID: \"470aba85-18f4-4acc-9e98-6233c2ae3c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:13.179291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.179207 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f7r\" (UniqueName: \"kubernetes.io/projected/cacd9cad-031b-4a85-a86d-a04eecb18d95-kube-api-access-z5f7r\") pod \"kas-connection-checker-685895c7c9-tkhfn\" (UID: \"cacd9cad-031b-4a85-a86d-a04eecb18d95\") " pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:13.179291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.179246 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rd8l\" (UniqueName: \"kubernetes.io/projected/70d94d6f-863f-4819-b36f-5d2c753471bd-kube-api-access-9rd8l\") pod \"kas-connection-checker-685895c7c9-ldcc4\" (UID: \"70d94d6f-863f-4819-b36f-5d2c753471bd\") " pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:13.179458 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.179437 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.180193 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.180169 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.180704 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.180685 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.181483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.181467 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:13.186109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.186091 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:13.187828 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.187805 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:13.187946 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.187931 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.188504 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.188488 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.191120 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.191100 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.242242 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.242179 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 20:36:13.242242 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.242179 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 20:36:13.242242 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.242205 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rtr46\"" Oct 30 20:36:13.242242 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.242185 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-57fvr\"" Oct 30 20:36:13.243100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243087 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 20:36:13.243178 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243158 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 20:36:13.243555 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243540 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 20:36:13.243588 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243557 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.243616 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243610 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 20:36:13.243703 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243677 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.243703 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243697 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.243874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243717 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vdhsx\"" Oct 30 20:36:13.243874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243725 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.243874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243733 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.243874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243781 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7m9rl\"" Oct 30 20:36:13.243874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.243807 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 20:36:13.244095 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.244078 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4kmkw\"" Oct 30 20:36:13.245379 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245363 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.245478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245393 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.245478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245464 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.245568 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245508 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 20:36:13.245759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245730 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.245876 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245809 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.245948 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.245935 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 20:36:13.246205 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.246186 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.246614 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.246595 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.247386 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.247371 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 20:36:13.250603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.250587 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rd8l\" (UniqueName: \"kubernetes.io/projected/70d94d6f-863f-4819-b36f-5d2c753471bd-kube-api-access-9rd8l\") pod \"kas-connection-checker-685895c7c9-ldcc4\" (UID: \"70d94d6f-863f-4819-b36f-5d2c753471bd\") " pod="kube-system/kas-connection-checker-685895c7c9-ldcc4" Oct 30 20:36:13.251012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.250842 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktzm\" (UniqueName: \"kubernetes.io/projected/470aba85-18f4-4acc-9e98-6233c2ae3c9c-kube-api-access-4ktzm\") pod \"kas-connection-checker-685895c7c9-sg6jp\" (UID: \"470aba85-18f4-4acc-9e98-6233c2ae3c9c\") " pod="kube-system/kas-connection-checker-685895c7c9-sg6jp" Oct 30 20:36:13.251383 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.251360 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f7r\" (UniqueName: \"kubernetes.io/projected/cacd9cad-031b-4a85-a86d-a04eecb18d95-kube-api-access-z5f7r\") pod \"kas-connection-checker-685895c7c9-tkhfn\" (UID: \"cacd9cad-031b-4a85-a86d-a04eecb18d95\") " pod="kube-system/kas-connection-checker-685895c7c9-tkhfn" Oct 30 20:36:13.275819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.275797 4988 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:36:13.280012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.279990 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd681674-072e-4dff-a369-9d956c32b960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:13.280083 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280028 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-netns\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280083 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280056 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxnr\" (UniqueName: \"kubernetes.io/projected/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-api-access-tnxnr\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.280159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280099 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:13.280159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280130 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.280159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280149 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-etc-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280254 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280222 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-ovn\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280254 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280251 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpbk\" (UniqueName: \"kubernetes.io/projected/e933dad7-264e-49fc-ae45-e6ba8ffad936-kube-api-access-fkpbk\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280321 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280289 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975f6dec-89de-4c6e-ab23-a665d5d76ced-agent-certs\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:36:13.280321 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280312 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfj6\" (UniqueName: \"kubernetes.io/projected/f24910a3-693b-4290-aec0-7539fe0115d7-kube-api-access-fsfj6\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.280422 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280391 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.280525 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280465 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-slash\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280525 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280473 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd681674-072e-4dff-a369-9d956c32b960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:13.280631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280547 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-config\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:13.280631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280582 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.280631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280612 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280783 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280644 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ngt\" (UniqueName: \"kubernetes.io/projected/8bfb7e64-5a1d-4470-9269-79be9505ebe8-kube-api-access-s2ngt\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.280783 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280662 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-snapshots\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.280783 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280679 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmm7\" (UniqueName: \"kubernetes.io/projected/f23150de-f251-49ea-a39c-9517da973316-kube-api-access-vhmm7\") pod \"migrator-66df5d496b-psb5h\" (UID: \"f23150de-f251-49ea-a39c-9517da973316\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:13.280783 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280776 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280798 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280835 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-bin\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280859 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-netd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280862 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-snapshots\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280882 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-env-overrides\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280877 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-config\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280918 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5qs\" (UniqueName: \"kubernetes.io/projected/af942c5a-661e-423c-b24b-a7a4a1db3bd9-kube-api-access-zk5qs\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.280998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.280958 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-bin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281047 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-config\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281073 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd681674-072e-4dff-a369-9d956c32b960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281108 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281201 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd681674-072e-4dff-a369-9d956c32b960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281209 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281231 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-script-lib\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281249 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdngs\" (UniqueName: \"kubernetes.io/projected/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-kube-api-access-sdngs\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:13.281298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281289 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-system-cni-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281323 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281362 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-kubelet\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281388 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-host\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281437 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975f6dec-89de-4c6e-ab23-a665d5d76ced-konnectivity-ca\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281498 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281519 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281531 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6xs\" (UniqueName: \"kubernetes.io/projected/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-kube-api-access-bv6xs\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.281655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281557 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhrc\" (UniqueName: \"kubernetes.io/projected/7ea62097-316d-49f5-bccf-dba48029bb6d-kube-api-access-pjhrc\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281663 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281693 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-kubelet-config\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281715 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-config\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281730 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-hostroot\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281745 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d68f309-a728-4000-a631-86ef2c50b3e0-serving-cert\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281776 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysconfig\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281833 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtmj\" (UniqueName: \"kubernetes.io/projected/ed47a78a-0c03-4f46-b274-5b1d162d1c73-kube-api-access-cvtmj\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281856 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.281913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281880 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-tuned\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281966 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-log-socket\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281985 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.281982 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-tuned\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282017 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-var-lib-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282042 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282045 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282063 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282086 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28kd\" (UniqueName: \"kubernetes.io/projected/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-kube-api-access-f28kd\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282130 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282236 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55m2\" (UniqueName: \"kubernetes.io/projected/cd681674-072e-4dff-a369-9d956c32b960-kube-api-access-l55m2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282256 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282276 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-multus-certs\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.282294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282293 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282308 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-tmp-dir\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282325 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282347 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb7e64-5a1d-4470-9269-79be9505ebe8-tmp-dir\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282363 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqr4\" (UniqueName: \"kubernetes.io/projected/3d68f309-a728-4000-a631-86ef2c50b3e0-kube-api-access-7hqr4\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282374 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-tmp-dir\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282385 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db8922f9-31fe-4afd-b227-313eecf0fcaa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282437 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-systemd-units\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282462 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bfb7e64-5a1d-4470-9269-79be9505ebe8-tmp-dir\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282465 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-node-log\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282523 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-conf-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282530 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282560 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282597 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcp7p\" (UniqueName: \"kubernetes.io/projected/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-kube-api-access-fcp7p\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282695 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aedd749-52bf-473f-86bb-f42e2cd459d0-crio-socket\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282712 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-sys-fs\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282750 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.282790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282791 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-kubernetes\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282864 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-netns\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282891 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-device-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282916 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-systemd\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282938 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-serving-cert\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.282966 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c365d189-b04a-4b12-8519-30c074796180-serving-cert\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283045 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqdc\" (UniqueName: \"kubernetes.io/projected/c365d189-b04a-4b12-8519-30c074796180-kube-api-access-frqdc\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283074 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-serving-cert\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283128 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c365d189-b04a-4b12-8519-30c074796180-serving-cert\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283140 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.283227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283210 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283240 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7092030c-f398-4180-9641-62b31400359a-host\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283261 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-systemd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283279 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283293 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovn-node-metrics-cert\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283361 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qrj\" (UniqueName: \"kubernetes.io/projected/f340b19f-a26a-4514-8785-f0714fc17696-kube-api-access-j5qrj\") pod \"volume-data-source-validator-58c9f5f46d-7sjnz\" (UID: \"f340b19f-a26a-4514-8785-f0714fc17696\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283380 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283395 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283432 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-dbus\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283450 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283491 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157b8177-38fc-4986-880f-e820ce5621b1-cert\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:13.283511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283513 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea62097-316d-49f5-bccf-dba48029bb6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283536 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283539 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cnibin\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283582 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskcx\" (UniqueName: \"kubernetes.io/projected/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab-kube-api-access-hskcx\") pod \"network-check-source-6f78d59f99-v2p5n\" (UID: \"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283606 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283624 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-k8s-cni-cncf-io\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283676 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-kubelet\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283700 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283742 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283799 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:13.283822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283818 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-registration-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283835 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-etc-kubernetes\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283854 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283879 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283897 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-run\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283913 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfb7e64-5a1d-4470-9269-79be9505ebe8-config-volume\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283934 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvwz\" (UniqueName: \"kubernetes.io/projected/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kube-api-access-ztvwz\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283967 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-sys\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.283991 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-tmp\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284019 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af942c5a-661e-423c-b24b-a7a4a1db3bd9-iptables-alerter-script\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284052 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.284091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284073 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-multus-daemon-config\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284093 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ea47fc6-4d87-4c4f-b623-801cfe65c047-tmp\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284122 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphqg\" (UniqueName: \"kubernetes.io/projected/4fa2e4fe-41cb-4690-bba9-206c3aef048a-kube-api-access-rphqg\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284141 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284165 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284180 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-lib-modules\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284196 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284218 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-stats-auth\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284290 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-hosts-file\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.284374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284315 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.284630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284401 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.284630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284453 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkl9\" (UniqueName: \"kubernetes.io/projected/1ea47fc6-4d87-4c4f-b623-801cfe65c047-kube-api-access-djkl9\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284523 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znd9n\" (UniqueName: \"kubernetes.io/projected/b3970765-0603-41f1-9802-a252c08fdce7-kube-api-access-znd9n\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:13.284630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284585 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.284630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284602 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-service-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.284630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284627 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7092030c-f398-4180-9641-62b31400359a-serviceca\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.284786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284644 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-default-certificate\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.284786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284694 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.284786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284723 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-system-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.284786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284753 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.284786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284761 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c365d189-b04a-4b12-8519-30c074796180-service-ca-bundle\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284795 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkvb\" (UniqueName: \"kubernetes.io/projected/7092030c-f398-4180-9641-62b31400359a-kube-api-access-cjkvb\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284816 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284834 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-key\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284852 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284876 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-socket-dir-parent\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284895 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-socket-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284911 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-var-lib-kubelet\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.284919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284913 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-default-certificate\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284949 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-cabundle\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284968 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.284979 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af942c5a-661e-423c-b24b-a7a4a1db3bd9-host-slash\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285040 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-cni-binary-copy\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285077 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-multus\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285107 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5jz\" (UniqueName: \"kubernetes.io/projected/157b8177-38fc-4986-880f-e820ce5621b1-kube-api-access-sg5jz\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285136 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285162 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285191 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v9h\" (UniqueName: \"kubernetes.io/projected/ee569c03-5200-472a-8348-382a2da0327b-kube-api-access-69v9h\") pod \"migrator-66df5d496b-8fmqx\" (UID: \"ee569c03-5200-472a-8348-382a2da0327b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:13.285233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285230 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-modprobe-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.285646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285269 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-conf\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.285646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285319 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-tmp\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.285646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285353 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aedd749-52bf-473f-86bb-f42e2cd459d0-data-volume\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.285646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285381 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-cnibin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.285646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285394 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c365d189-b04a-4b12-8519-30c074796180-tmp\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.285646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.285530 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aedd749-52bf-473f-86bb-f42e2cd459d0-data-volume\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.297847 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.297822 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.369489 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369451 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 20:36:13.369652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369623 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.369719 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369662 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.369719 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369704 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369716 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369736 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8v9jn\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369716 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369777 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369778 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369786 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:36:13.369812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369790 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369831 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369916 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369968 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369971 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369972 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.369968 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370003 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370006 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370032 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2mcfh\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370084 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370135 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370142 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370144 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370169 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370187 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370199 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370211 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370242 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370244 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370246 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370405 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370242 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:36:13.370425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370265 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370471 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370484 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370504 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370548 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v74vw\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370611 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370724 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370766 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/975f6dec-89de-4c6e-ab23-a665d5d76ced-agent-certs\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:36:13.370940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370927 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qdrm5\"" Oct 30 20:36:13.371170 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370931 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 20:36:13.371170 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370961 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 20:36:13.371170 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.370973 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mk8vz\"" Oct 30 20:36:13.371280 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.371188 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qrj\" (UniqueName: \"kubernetes.io/projected/f340b19f-a26a-4514-8785-f0714fc17696-kube-api-access-j5qrj\") pod \"volume-data-source-validator-58c9f5f46d-7sjnz\" (UID: \"f340b19f-a26a-4514-8785-f0714fc17696\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-7sjnz" Oct 30 20:36:13.371280 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.371190 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-env-overrides\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.371280 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.371206 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28kd\" (UniqueName: \"kubernetes.io/projected/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-kube-api-access-f28kd\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.371991 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.371969 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pdv7w\"" Oct 30 20:36:13.372100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.372004 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-config\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.372100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.372049 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/975f6dec-89de-4c6e-ab23-a665d5d76ced-konnectivity-ca\") pod \"konnectivity-agent-l2csm\" (UID: \"975f6dec-89de-4c6e-ab23-a665d5d76ced\") " pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:36:13.372185 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.372146 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 20:36:13.372222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.372198 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 20:36:13.373484 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.373470 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db8922f9-31fe-4afd-b227-313eecf0fcaa-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:13.373535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.373511 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.373566 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.373535 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.374737 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.374719 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-stats-auth\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.374737 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.374731 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-multus-daemon-config\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.374852 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.374724 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/157b8177-38fc-4986-880f-e820ce5621b1-cert\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:13.374852 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.374758 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfb7e64-5a1d-4470-9269-79be9505ebe8-config-volume\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.375759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.375742 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.375819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.375778 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f24910a3-693b-4290-aec0-7539fe0115d7-cni-binary-copy\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.375819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.375796 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-cabundle\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:13.375877 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.375860 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed47a78a-0c03-4f46-b274-5b1d162d1c73-signing-key\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:13.378006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.377992 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mx55q\"" Oct 30 20:36:13.378203 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.378189 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:36:13.378371 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.378355 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:36:13.378432 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.378405 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b4p7g\"" Oct 30 20:36:13.378497 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.378450 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k4gm7\"" Oct 30 20:36:13.381563 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.381547 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-config\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.382892 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.382879 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l8lvp\"" Oct 30 20:36:13.382933 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.382882 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hqxsv\"" Oct 30 20:36:13.382995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.382980 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2nx5p\"" Oct 30 20:36:13.383041 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.382983 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rgvdg\"" Oct 30 20:36:13.383041 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.382999 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6lzrx\"" Oct 30 20:36:13.383041 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383001 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9pwzv\"" Oct 30 20:36:13.383041 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.382977 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k65fh\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383057 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383101 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383149 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383161 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383183 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383188 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383210 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.383216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383217 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.383487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383476 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:36:13.383532 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383506 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 20:36:13.383698 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383687 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:36:13.383774 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.383759 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovn-node-metrics-cert\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.384107 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.384085 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55m2\" (UniqueName: \"kubernetes.io/projected/cd681674-072e-4dff-a369-9d956c32b960-kube-api-access-l55m2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv\" (UID: \"cd681674-072e-4dff-a369-9d956c32b960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" Oct 30 20:36:13.384351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.384337 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af942c5a-661e-423c-b24b-a7a4a1db3bd9-iptables-alerter-script\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.384769 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.384756 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cn78n\"" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385129 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xntv\" (UniqueName: \"kubernetes.io/projected/5f48b68d-7d92-4417-915f-785bb3e457ab-kube-api-access-5xntv\") pod \"network-check-target-jdq2q\" (UID: \"5f48b68d-7d92-4417-915f-785bb3e457ab\") " pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385195 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskcx\" (UniqueName: \"kubernetes.io/projected/7b81cd8d-9d74-4e6d-bda5-7b26aa16feab-kube-api-access-hskcx\") pod \"network-check-source-6f78d59f99-v2p5n\" (UID: \"7b81cd8d-9d74-4e6d-bda5-7b26aa16feab\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-v2p5n" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385318 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxnr\" (UniqueName: \"kubernetes.io/projected/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-api-access-tnxnr\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385342 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385390 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfj6\" (UniqueName: \"kubernetes.io/projected/f24910a3-693b-4290-aec0-7539fe0115d7-kube-api-access-fsfj6\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385395 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385566 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6xs\" (UniqueName: \"kubernetes.io/projected/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-kube-api-access-bv6xs\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385578 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtmj\" (UniqueName: \"kubernetes.io/projected/ed47a78a-0c03-4f46-b274-5b1d162d1c73-kube-api-access-cvtmj\") pod \"service-ca-5fffc97bdf-h4ncf\" (UID: \"ed47a78a-0c03-4f46-b274-5b1d162d1c73\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h4ncf" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385569 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqdc\" (UniqueName: \"kubernetes.io/projected/c365d189-b04a-4b12-8519-30c074796180-kube-api-access-frqdc\") pod \"insights-operator-7c4986d449-tmxhb\" (UID: \"c365d189-b04a-4b12-8519-30c074796180\") " pod="openshift-insights/insights-operator-7c4986d449-tmxhb" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385795 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-system-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385809 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ngt\" (UniqueName: \"kubernetes.io/projected/8bfb7e64-5a1d-4470-9269-79be9505ebe8-kube-api-access-s2ngt\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385836 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385866 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-socket-dir-parent\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385898 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-socket-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385869 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-system-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385927 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-var-lib-kubelet\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385930 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvwz\" (UniqueName: \"kubernetes.io/projected/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kube-api-access-ztvwz\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.385948 4988 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385920 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhrc\" (UniqueName: \"kubernetes.io/projected/7ea62097-316d-49f5-bccf-dba48029bb6d-kube-api-access-pjhrc\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385954 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-socket-dir-parent\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385976 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af942c5a-661e-423c-b24b-a7a4a1db3bd9-host-slash\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385984 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-socket-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385989 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-var-lib-kubelet\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.386004 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.885989546 +0000 UTC m=+1.993485435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386020 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af942c5a-661e-423c-b24b-a7a4a1db3bd9-host-slash\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.386012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.385993 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znd9n\" (UniqueName: \"kubernetes.io/projected/b3970765-0603-41f1-9802-a252c08fdce7-kube-api-access-znd9n\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386055 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-multus\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386084 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386109 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386086 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-multus\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386135 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-modprobe-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.386143 4988 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386158 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-conf\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386168 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5jz\" (UniqueName: \"kubernetes.io/projected/157b8177-38fc-4986-880f-e820ce5621b1-kube-api-access-sg5jz\") pod \"ingress-canary-s6pl6\" (UID: \"157b8177-38fc-4986-880f-e820ce5621b1\") " pod="openshift-ingress-canary/ingress-canary-s6pl6" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386185 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-cnibin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.386199 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.88618327 +0000 UTC m=+1.993679177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386229 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-cnibin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386230 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-netns\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386259 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-netns\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386271 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-modprobe-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386278 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-conf\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386280 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386312 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386332 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-etc-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386349 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-ovn\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386385 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-slash\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386387 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-etc-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386403 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386438 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386444 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-bin\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386446 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-ovn\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386461 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-netd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386463 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-bin\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386480 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-cni-netd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386480 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-bin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386445 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-slash\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386511 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386503 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-cni-bin\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386540 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-system-cni-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386561 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-kubelet\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386577 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-host\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.386562 4988 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386589 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-system-cni-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386592 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-kubelet\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386607 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-kubelet-config\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.386633 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.886619651 +0000 UTC m=+1.994115570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : configmap "kube-rbac-proxy" not found Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386648 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-host\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386651 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-hostroot\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386666 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-kubelet-config\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386678 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysconfig\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386699 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386705 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-hostroot\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386716 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-log-socket\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386749 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-var-lib-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386760 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-log-socket\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386769 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386767 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysconfig\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386801 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-var-lib-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386808 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386825 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386845 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-multus-certs\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386802 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-sysctl-d\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386865 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-systemd-units\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386882 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-systemd-units\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386888 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386902 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-multus-certs\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386910 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-node-log\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386936 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-node-log\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386936 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-conf-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386953 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-conf-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.386931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386964 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aedd749-52bf-473f-86bb-f42e2cd459d0-crio-socket\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386996 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-sys-fs\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.386999 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5aedd749-52bf-473f-86bb-f42e2cd459d0-crio-socket\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387020 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-kubernetes\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387039 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-netns\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387054 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-device-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387067 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-sys-fs\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387068 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-systemd\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387091 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-kubernetes\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387110 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-netns\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387118 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7092030c-f398-4180-9641-62b31400359a-host\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387133 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-device-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387147 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-systemd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387162 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-etc-systemd\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387166 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7092030c-f398-4180-9641-62b31400359a-host\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387178 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387208 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-systemd\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387207 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387242 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-dbus\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387271 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cnibin\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387304 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-cnibin\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387273 4988 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387352 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387356 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387314 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-multus-cni-dir\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387314 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-dbus\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387371 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs podName:3efc8cc8-17bc-426c-84e1-43e4d7888e1c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.887354168 +0000 UTC m=+1.994850076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs") pod "router-default-789f8dddc5-m74zk" (UID: "3efc8cc8-17bc-426c-84e1-43e4d7888e1c") : secret "router-metrics-certs-default" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387405 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-k8s-cni-cncf-io\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387451 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-run-k8s-cni-cncf-io\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387453 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-kubelet\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387468 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387476 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387478 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387489 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-host-var-lib-kubelet\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387517 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa2e4fe-41cb-4690-bba9-206c3aef048a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387538 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387567 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-registration-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387596 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-etc-kubernetes\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387602 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387619 4988 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387637 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24910a3-693b-4290-aec0-7539fe0115d7-etc-kubernetes\") pod \"multus-r7m68\" (UID: \"f24910a3-693b-4290-aec0-7539fe0115d7\") " pod="openshift-multus/multus-r7m68" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387639 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2-registration-dir\") pod \"aws-ebs-csi-driver-node-ptb9c\" (UID: \"6f7257f1-1a1e-4029-8c65-f9c8e5ec72f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptb9c" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387621 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-run\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387664 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.887650089 +0000 UTC m=+1.995145987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : secret "metrics-daemon-secret" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387661 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-run\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387692 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387694 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-sys\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387724 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387725 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-sys\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387746 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387771 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-lib-modules\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387796 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387830 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-hosts-file\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387843 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-host-run-ovn-kubernetes\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387862 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387834 4988 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387896 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5b875558-4e2e-4eaf-9dc6-7861f1a95be0-hosts-file\") pod \"node-resolver-7kw6v\" (UID: \"5b875558-4e2e-4eaf-9dc6-7861f1a95be0\") " pod="openshift-dns/node-resolver-7kw6v" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387863 4988 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387929 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.887914092 +0000 UTC m=+1.995410014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387934 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e933dad7-264e-49fc-ae45-e6ba8ffad936-run-openvswitch\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.387936 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1ea47fc6-4d87-4c4f-b623-801cfe65c047-lib-modules\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.387969 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.887955006 +0000 UTC m=+1.995450905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388046 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388113 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hbdn4\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388123 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hxg8n\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388139 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-service-ca-bundle\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388310 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-x2fjq\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388704 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-79tcj\"" Oct 30 20:36:13.389065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.388926 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcp7p\" (UniqueName: \"kubernetes.io/projected/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-kube-api-access-fcp7p\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.390952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.390842 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.391059 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.391038 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.391371 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.391358 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqr4\" (UniqueName: \"kubernetes.io/projected/3d68f309-a728-4000-a631-86ef2c50b3e0-kube-api-access-7hqr4\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.391564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.391550 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e933dad7-264e-49fc-ae45-e6ba8ffad936-ovnkube-script-lib\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.392239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.392223 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d68f309-a728-4000-a631-86ef2c50b3e0-serving-cert\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.392345 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.392332 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.393220 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.393206 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.394074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.394059 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.394118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.394099 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ea62097-316d-49f5-bccf-dba48029bb6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.394376 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.394361 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpbk\" (UniqueName: \"kubernetes.io/projected/e933dad7-264e-49fc-ae45-e6ba8ffad936-kube-api-access-fkpbk\") pod \"ovnkube-node-27nvb\" (UID: \"e933dad7-264e-49fc-ae45-e6ba8ffad936\") " pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:13.394503 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.394490 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.395250 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.395198 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7092030c-f398-4180-9641-62b31400359a-serviceca\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.396015 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.395995 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.396224 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396209 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:13.396293 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396226 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:13.396293 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396285 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.896270371 +0000 UTC m=+2.003766279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:13.396386 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396368 4988 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 20:36:13.396470 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396457 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:13.396520 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396473 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:13.396520 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.396492 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qrh4r\"" Oct 30 20:36:13.396520 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396497 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert podName:db8922f9-31fe-4afd-b227-313eecf0fcaa nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.896458392 +0000 UTC m=+2.003954302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zbpwc" (UID: "db8922f9-31fe-4afd-b227-313eecf0fcaa") : secret "networking-console-plugin-cert" not found Oct 30 20:36:13.396621 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.396526 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:13.896513717 +0000 UTC m=+2.004009615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:13.396621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.396585 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5qs\" (UniqueName: \"kubernetes.io/projected/af942c5a-661e-423c-b24b-a7a4a1db3bd9-kube-api-access-zk5qs\") pod \"iptables-alerter-8ngvj\" (UID: \"af942c5a-661e-423c-b24b-a7a4a1db3bd9\") " pod="openshift-network-operator/iptables-alerter-8ngvj" Oct 30 20:36:13.397947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.397930 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.420118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.420091 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:36:13.420118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.420090 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-csvtm\"" Oct 30 20:36:13.421513 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.421323 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmm7\" (UniqueName: \"kubernetes.io/projected/f23150de-f251-49ea-a39c-9517da973316-kube-api-access-vhmm7\") pod \"migrator-66df5d496b-psb5h\" (UID: \"f23150de-f251-49ea-a39c-9517da973316\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-psb5h" Oct 30 20:36:13.421513 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.421378 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphqg\" (UniqueName: \"kubernetes.io/projected/4fa2e4fe-41cb-4690-bba9-206c3aef048a-kube-api-access-rphqg\") pod \"multus-additional-cni-plugins-4fv75\" (UID: \"4fa2e4fe-41cb-4690-bba9-206c3aef048a\") " pod="openshift-multus/multus-additional-cni-plugins-4fv75" Oct 30 20:36:13.421513 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.421461 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v9h\" (UniqueName: \"kubernetes.io/projected/ee569c03-5200-472a-8348-382a2da0327b-kube-api-access-69v9h\") pod \"migrator-66df5d496b-8fmqx\" (UID: \"ee569c03-5200-472a-8348-382a2da0327b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8fmqx" Oct 30 20:36:13.421755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.421520 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") pod \"periodic-gathering-jfr7m-mw5r6\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:36:13.423389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.422062 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkl9\" (UniqueName: \"kubernetes.io/projected/1ea47fc6-4d87-4c4f-b623-801cfe65c047-kube-api-access-djkl9\") pod \"tuned-wfcwm\" (UID: \"1ea47fc6-4d87-4c4f-b623-801cfe65c047\") " pod="openshift-cluster-node-tuning-operator/tuned-wfcwm" Oct 30 20:36:13.423389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.422147 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkvb\" (UniqueName: \"kubernetes.io/projected/7092030c-f398-4180-9641-62b31400359a-kube-api-access-cjkvb\") pod \"node-ca-7dvbw\" (UID: \"7092030c-f398-4180-9641-62b31400359a\") " pod="openshift-image-registry/node-ca-7dvbw" Oct 30 20:36:13.423389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.422165 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdngs\" (UniqueName: \"kubernetes.io/projected/0a3af4a4-5731-41d0-8f14-cbe1543de1f7-kube-api-access-sdngs\") pod \"service-ca-operator-544d749c8c-88htp\" (UID: \"0a3af4a4-5731-41d0-8f14-cbe1543de1f7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" Oct 30 20:36:13.423389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.422513 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 20:36:13.423389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.423024 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 20:36:13.425463 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.425444 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ed71fea-5f6c-4ba4-be09-b524c1abfc4a-original-pull-secret\") pod \"global-pull-secret-syncer-2kjtc\" (UID: \"6ed71fea-5f6c-4ba4-be09-b524c1abfc4a\") " pod="kube-system/global-pull-secret-syncer-2kjtc" Oct 30 20:36:13.425547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.425470 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.431343 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.431322 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d68f309-a728-4000-a631-86ef2c50b3e0-trusted-ca\") pod \"console-operator-7cd56b8dbc-2lsbc\" (UID: \"3d68f309-a728-4000-a631-86ef2c50b3e0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:13.433284 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.433261 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.479965 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.479932 4988 scope.go:122] "RemoveContainer" containerID="48ac13495f724b75a2bf513a70f9488b8a8ff404a2cdd942348ba688a0f89557" Oct 30 20:36:13.489119 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.489088 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:13.764160 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.764068 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:13.893124 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893089 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:13.893309 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893142 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.893309 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893171 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:13.893309 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893212 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:13.893309 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893240 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:13.893309 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893258 4988 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893318 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.893302271 +0000 UTC m=+3.000798161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : secret "metrics-daemon-secret" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893323 4988 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893352 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893372 4988 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893379 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.893366525 +0000 UTC m=+3.000862417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893323 4988 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893440 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.893400353 +0000 UTC m=+3.000896246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893326 4988 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893461 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.893452529 +0000 UTC m=+3.000948419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.893498 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.8934752 +0000 UTC m=+3.000971133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:13.893606 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893545 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.893923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893814 4988 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:36:13.893923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.893907 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5aedd749-52bf-473f-86bb-f42e2cd459d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:13.896119 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.896098 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3efc8cc8-17bc-426c-84e1-43e4d7888e1c-metrics-certs\") pod \"router-default-789f8dddc5-m74zk\" (UID: \"3efc8cc8-17bc-426c-84e1-43e4d7888e1c\") " pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:13.959322 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.959281 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:13.994870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.994835 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:13.995026 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.994879 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:13.995026 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.994950 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:13.995144 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.995098 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:13.995144 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.995116 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:13.995144 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.995117 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:13.995144 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.995136 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:13.995315 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.995169 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.995156856 +0000 UTC m=+3.102652750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:13.995315 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:13.995182 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:14.99517684 +0000 UTC m=+3.102672729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:13.997328 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:13.997302 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db8922f9-31fe-4afd-b227-313eecf0fcaa-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zbpwc\" (UID: \"db8922f9-31fe-4afd-b227-313eecf0fcaa\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:14.076754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.076682 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:36:14.076893 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.076798 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" event={"ID":"3d68f309-a728-4000-a631-86ef2c50b3e0","Type":"ContainerStarted","Data":"84a8575a84d8ce6b4aadb360d7cabe6124b4e8b40558b543e2e610910e5e8158"} Oct 30 20:36:14.080453 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.080432 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:14.082553 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.082536 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" Oct 30 20:36:14.251077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.251048 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7dvbw_7092030c-f398-4180-9641-62b31400359a/node-ca/0.log" Oct 30 20:36:14.399784 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.399749 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:14.544261 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.544217 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:14.574826 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.574792 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zbpwc"] Oct 30 20:36:14.579372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.579343 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-789f8dddc5-m74zk"] Oct 30 20:36:14.580019 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:14.579980 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8922f9_31fe_4afd_b227_313eecf0fcaa.slice/crio-319ba07068cbc89767aa4c3e1dd86b7ccad94303d1250f5cc7b8736beca0174c WatchSource:0}: Error finding container 319ba07068cbc89767aa4c3e1dd86b7ccad94303d1250f5cc7b8736beca0174c: Status 404 returned error can't find the container with id 319ba07068cbc89767aa4c3e1dd86b7ccad94303d1250f5cc7b8736beca0174c Oct 30 20:36:14.582228 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.582213 4988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:36:14.630104 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:14.630067 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efc8cc8_17bc_426c_84e1_43e4d7888e1c.slice/crio-ec382f45bb1e47c858b9c4fcbb934cd5b3a9e6b40c6d1cac7a154b6139330d1c WatchSource:0}: Error finding container ec382f45bb1e47c858b9c4fcbb934cd5b3a9e6b40c6d1cac7a154b6139330d1c: Status 404 returned error can't find the container with id ec382f45bb1e47c858b9c4fcbb934cd5b3a9e6b40c6d1cac7a154b6139330d1c Oct 30 20:36:14.691029 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.690970 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:14.835720 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.835676 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:14.851339 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.851299 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6pl6_157b8177-38fc-4986-880f-e820ce5621b1/serve-healthcheck-canary/0.log" Oct 30 20:36:14.903923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.903887 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:14.904103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.903933 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:14.904103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.903956 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:14.904103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.903990 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:14.904103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.904014 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:14.904103 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904069 4988 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904108 4988 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904137 4988 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904139 4988 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904141 4988 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904146 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.904124797 +0000 UTC m=+5.011620711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904258 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.904244223 +0000 UTC m=+5.011740116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904270 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.904263341 +0000 UTC m=+5.011759231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904283 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.90427695 +0000 UTC m=+5.011772839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : secret "metrics-daemon-secret" not found Oct 30 20:36:14.904348 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:14.904292 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:16.904287497 +0000 UTC m=+5.011783387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:14.985035 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:14.984948 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.005110 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.005077 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:15.005247 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.005117 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:15.005247 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:15.005238 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:15.005336 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:15.005245 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:15.005336 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:15.005269 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:15.005336 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:15.005254 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:15.005530 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:15.005340 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:17.005324038 +0000 UTC m=+5.112819952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:15.005530 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:15.005359 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:17.005350714 +0000 UTC m=+5.112846605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:15.052433 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.052380 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8fmqx_ee569c03-5200-472a-8348-382a2da0327b/migrator/0.log" Oct 30 20:36:15.082363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.082328 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-789f8dddc5-m74zk" event={"ID":"3efc8cc8-17bc-426c-84e1-43e4d7888e1c","Type":"ContainerStarted","Data":"4a42dfb35613ca26a6cb7d22cf1a22227f8001633e23e219d3def38f35e9b703"} Oct 30 20:36:15.082550 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.082431 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-789f8dddc5-m74zk" event={"ID":"3efc8cc8-17bc-426c-84e1-43e4d7888e1c","Type":"ContainerStarted","Data":"ec382f45bb1e47c858b9c4fcbb934cd5b3a9e6b40c6d1cac7a154b6139330d1c"} Oct 30 20:36:15.083819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.083789 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" event={"ID":"db8922f9-31fe-4afd-b227-313eecf0fcaa","Type":"ContainerStarted","Data":"319ba07068cbc89767aa4c3e1dd86b7ccad94303d1250f5cc7b8736beca0174c"} Oct 30 20:36:15.150351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.150293 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.251655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.251585 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-psb5h_f23150de-f251-49ea-a39c-9517da973316/migrator/0.log" Oct 30 20:36:15.291223 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.291175 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.453333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.453301 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv_cd681674-072e-4dff-a369-9d956c32b960/kube-storage-version-migrator-operator/0.log" Oct 30 20:36:15.637611 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.637576 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.782226 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.782186 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.947841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:15.947762 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:16.009903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.009867 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:16.081280 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.081243 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:16.084508 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.084479 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:16.088795 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.088772 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-789f8dddc5-m74zk" Oct 30 20:36:16.152559 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.152530 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:16.266271 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.266210 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:16.416105 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.416070 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:16.564021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.563939 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:16.579232 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.579204 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l2csm" Oct 30 20:36:16.608000 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.607965 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:16.608179 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.608111 4988 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:36:16.611836 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.611812 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2lsbc" Oct 30 20:36:16.752724 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.752694 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:16.922383 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.922347 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.922398 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.922451 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.922487 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.922515 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922529 4988 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922594 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:20.922579454 +0000 UTC m=+9.030075344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:16.922600 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922598 4988 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922631 4988 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922654 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:20.922641188 +0000 UTC m=+9.030137103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922598 4988 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922673 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls podName:8bfb7e64-5a1d-4470-9269-79be9505ebe8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:20.922660816 +0000 UTC m=+9.030156707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls") pod "dns-default-8jh5r" (UID: "8bfb7e64-5a1d-4470-9269-79be9505ebe8") : secret "dns-default-metrics-tls" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922694 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls podName:2b9522ce-81d5-40d6-8ac2-ef99be4474c8 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:20.922687463 +0000 UTC m=+9.030183352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vzmj2" (UID: "2b9522ce-81d5-40d6-8ac2-ef99be4474c8") : secret "samples-operator-tls" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922529 4988 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 20:36:16.922922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:16.922719 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:20.922714312 +0000 UTC m=+9.030210201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : secret "metrics-daemon-secret" not found Oct 30 20:36:16.945285 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:16.945256 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:17.023800 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:17.023756 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:17.023800 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:17.023804 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:17.024257 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:17.023909 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:17.024257 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:17.023929 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85df8d78bd-v8dx4: secret "image-registry-tls" not found Oct 30 20:36:17.024257 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:17.023952 4988 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 20:36:17.024257 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:17.023968 4988 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587bd9b797-w6c54: secret "image-registry-tls" not found Oct 30 20:36:17.024257 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:17.023980 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls podName:be31e37c-2894-420c-84cc-16fa0776f92c nodeName:}" failed. No retries permitted until 2026-10-30 20:36:21.023965676 +0000 UTC m=+9.131461566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls") pod "image-registry-85df8d78bd-v8dx4" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c") : secret "image-registry-tls" not found Oct 30 20:36:17.024257 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:17.024007 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls podName:391e9838-62c0-4d2d-8d8e-29d7493eff20 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:21.023995458 +0000 UTC m=+9.131491352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls") pod "image-registry-587bd9b797-w6c54" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20") : secret "image-registry-tls" not found Oct 30 20:36:17.093662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:17.093624 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zbpwc" event={"ID":"db8922f9-31fe-4afd-b227-313eecf0fcaa","Type":"ContainerStarted","Data":"79bdc039fdb9a318ce21cb2cb2a3f19790931bbc10ff37acb09fc3a0732fffc1"} Oct 30 20:36:17.145352 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:17.145321 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:17.151468 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:17.151441 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:17.300702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:17.300596 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:18.364281 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:18.364245 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:18.548477 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:18.548435 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:18.718131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:18.718076 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:18.866239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:18.866209 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:19.016087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:19.016005 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:19.160301 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:19.160255 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:20.346087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.346052 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:20.546113 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.546076 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:20.752883 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.752849 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:20.946779 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.946743 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:20.966894 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.966861 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:20.967025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.966901 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:20.967025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.966942 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:20.967025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.966972 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:20.967025 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:20.967003 4988 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:36:20.967225 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:20.967062 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls podName:5aedd749-52bf-473f-86bb-f42e2cd459d0 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:28.967047177 +0000 UTC m=+17.074543067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdlwf" (UID: "5aedd749-52bf-473f-86bb-f42e2cd459d0") : secret "insights-runtime-extractor-tls" not found Oct 30 20:36:20.967225 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:20.967067 4988 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:20.967225 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:20.967126 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls podName:7ea62097-316d-49f5-bccf-dba48029bb6d nodeName:}" failed. No retries permitted until 2026-10-30 20:36:28.967110636 +0000 UTC m=+17.074606540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tvpjf" (UID: "7ea62097-316d-49f5-bccf-dba48029bb6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 20:36:20.967225 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.967148 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:20.967520 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:20.967233 4988 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 20:36:20.967520 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:20.967269 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs podName:b3970765-0603-41f1-9802-a252c08fdce7 nodeName:}" failed. No retries permitted until 2026-10-30 20:36:28.967258137 +0000 UTC m=+17.074754040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs") pod "network-metrics-daemon-7bt9s" (UID: "b3970765-0603-41f1-9802-a252c08fdce7") : secret "metrics-daemon-secret" not found Oct 30 20:36:20.969421 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.969387 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bfb7e64-5a1d-4470-9269-79be9505ebe8-metrics-tls\") pod \"dns-default-8jh5r\" (UID: \"8bfb7e64-5a1d-4470-9269-79be9505ebe8\") " pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:20.969560 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.969538 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b9522ce-81d5-40d6-8ac2-ef99be4474c8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vzmj2\" (UID: \"2b9522ce-81d5-40d6-8ac2-ef99be4474c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:20.976335 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.976318 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" Oct 30 20:36:20.989312 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:20.989284 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:21.067636 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.067606 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:21.067761 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.067642 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:21.069927 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.069870 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"image-registry-587bd9b797-w6c54\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:21.070323 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.070297 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"image-registry-85df8d78bd-v8dx4\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:21.145681 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.145642 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:21.281914 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.281840 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:21.290361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.288331 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:21.353833 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.353797 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2"] Oct 30 20:36:21.406826 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.406795 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jh5r"] Oct 30 20:36:21.410473 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:21.410437 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfb7e64_5a1d_4470_9269_79be9505ebe8.slice/crio-1d96c25ff8f203272a19a72724ae2e897f4bff160c01641af0936b7e41def5c2 WatchSource:0}: Error finding container 1d96c25ff8f203272a19a72724ae2e897f4bff160c01641af0936b7e41def5c2: Status 404 returned error can't find the container with id 1d96c25ff8f203272a19a72724ae2e897f4bff160c01641af0936b7e41def5c2 Oct 30 20:36:21.663709 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.663668 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-587bd9b797-w6c54"] Oct 30 20:36:21.667239 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:21.667210 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391e9838_62c0_4d2d_8d8e_29d7493eff20.slice/crio-740132b84afa0343c5794b9b4fa7fde34a7feff826d34f68d4a37cb01f772ce5 WatchSource:0}: Error finding container 740132b84afa0343c5794b9b4fa7fde34a7feff826d34f68d4a37cb01f772ce5: Status 404 returned error can't find the container with id 740132b84afa0343c5794b9b4fa7fde34a7feff826d34f68d4a37cb01f772ce5 Oct 30 20:36:21.714077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:21.714047 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85df8d78bd-v8dx4"] Oct 30 20:36:21.717952 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:21.717925 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe31e37c_2894_420c_84cc_16fa0776f92c.slice/crio-5b0495c216aafb871df12109301300115ad59e1a51fac18e3de2d747fa99cb50 WatchSource:0}: Error finding container 5b0495c216aafb871df12109301300115ad59e1a51fac18e3de2d747fa99cb50: Status 404 returned error can't find the container with id 5b0495c216aafb871df12109301300115ad59e1a51fac18e3de2d747fa99cb50 Oct 30 20:36:22.112927 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.112824 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" event={"ID":"be31e37c-2894-420c-84cc-16fa0776f92c","Type":"ContainerStarted","Data":"8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2"} Oct 30 20:36:22.112927 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.112881 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" event={"ID":"be31e37c-2894-420c-84cc-16fa0776f92c","Type":"ContainerStarted","Data":"5b0495c216aafb871df12109301300115ad59e1a51fac18e3de2d747fa99cb50"} Oct 30 20:36:22.114440 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.114384 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" event={"ID":"391e9838-62c0-4d2d-8d8e-29d7493eff20","Type":"ContainerStarted","Data":"07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2"} Oct 30 20:36:22.114534 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.114453 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" event={"ID":"391e9838-62c0-4d2d-8d8e-29d7493eff20","Type":"ContainerStarted","Data":"740132b84afa0343c5794b9b4fa7fde34a7feff826d34f68d4a37cb01f772ce5"} Oct 30 20:36:22.115523 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.115501 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jh5r" event={"ID":"8bfb7e64-5a1d-4470-9269-79be9505ebe8","Type":"ContainerStarted","Data":"1d96c25ff8f203272a19a72724ae2e897f4bff160c01641af0936b7e41def5c2"} Oct 30 20:36:22.116655 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.116633 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" event={"ID":"2b9522ce-81d5-40d6-8ac2-ef99be4474c8","Type":"ContainerStarted","Data":"b72f806dd6cf7708ed1f137c12c51e2edf201e922044f9c6d09b61aeb77bd533"} Oct 30 20:36:22.347823 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.347785 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:22.552979 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.552943 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:22.744534 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.744493 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:22.945316 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:22.945269 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:23.122503 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:23.122472 4988 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:36:23.124446 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:23.123561 4988 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:36:23.134843 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:23.134814 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:23.135053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:23.135036 4988 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:36:23.147235 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:23.147203 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:23.543999 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:23.543957 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:24.131468 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:24.131438 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jh5r" event={"ID":"8bfb7e64-5a1d-4470-9269-79be9505ebe8","Type":"ContainerStarted","Data":"f9c38ba7338bd8cda38e5913f9858ba4c063b56a248e9c5f8a961b4717ff5ea3"} Oct 30 20:36:24.131786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:24.131515 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jh5r" event={"ID":"8bfb7e64-5a1d-4470-9269-79be9505ebe8","Type":"ContainerStarted","Data":"e1f4c6478a6eaf7e0dcdb3ca969b238f81ee1c1ced51e39532fdcec9d3c43ab0"} Oct 30 20:36:24.131786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:24.131578 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:25.137305 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:25.137267 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" event={"ID":"2b9522ce-81d5-40d6-8ac2-ef99be4474c8","Type":"ContainerStarted","Data":"2c2bccb02ec9c30fa0eb8de9c6452b9c08d6f44084a8cc56d13f953ad34ad19f"} Oct 30 20:36:25.137990 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:25.137326 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vzmj2" event={"ID":"2b9522ce-81d5-40d6-8ac2-ef99be4474c8","Type":"ContainerStarted","Data":"e3a04284871a9619cae7e5c8d5fd1be9db5c0d37405ff5a54b31a23719e2691c"} Oct 30 20:36:25.617438 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:25.617337 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:26.079681 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:26.079646 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:27.251606 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:27.251571 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:29.047529 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.047487 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:29.047529 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.047531 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:29.047957 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.047623 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:29.049830 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.049798 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5aedd749-52bf-473f-86bb-f42e2cd459d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdlwf\" (UID: \"5aedd749-52bf-473f-86bb-f42e2cd459d0\") " pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:29.049963 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.049806 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3970765-0603-41f1-9802-a252c08fdce7-metrics-certs\") pod \"network-metrics-daemon-7bt9s\" (UID: \"b3970765-0603-41f1-9802-a252c08fdce7\") " pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:29.050061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.050038 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ea62097-316d-49f5-bccf-dba48029bb6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tvpjf\" (UID: \"7ea62097-316d-49f5-bccf-dba48029bb6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:29.080845 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.080818 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" Oct 30 20:36:29.086375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.086349 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bt9s" Oct 30 20:36:29.092035 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.092009 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdlwf" Oct 30 20:36:29.520233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.520192 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf"] Oct 30 20:36:29.524074 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:29.524048 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea62097_316d_49f5_bccf_dba48029bb6d.slice/crio-a26976f9d3eecd363b27e91e1d8e9bca4130ae9445d61f143c76b383981d7bec WatchSource:0}: Error finding container a26976f9d3eecd363b27e91e1d8e9bca4130ae9445d61f143c76b383981d7bec: Status 404 returned error can't find the container with id a26976f9d3eecd363b27e91e1d8e9bca4130ae9445d61f143c76b383981d7bec Oct 30 20:36:29.562152 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:29.562124 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3970765_0603_41f1_9802_a252c08fdce7.slice/crio-e99fb4df29cd7056b23deb20b0025851e8d49c25c17fa085965b0b403ed5e31e WatchSource:0}: Error finding container e99fb4df29cd7056b23deb20b0025851e8d49c25c17fa085965b0b403ed5e31e: Status 404 returned error can't find the container with id e99fb4df29cd7056b23deb20b0025851e8d49c25c17fa085965b0b403ed5e31e Oct 30 20:36:29.570012 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:29.569963 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aedd749_52bf_473f_86bb_f42e2cd459d0.slice/crio-d2c08238389cc3f9c359c57e146f0a92e564e56a73c174c8e69808341f9391da WatchSource:0}: Error finding container d2c08238389cc3f9c359c57e146f0a92e564e56a73c174c8e69808341f9391da: Status 404 returned error can't find the container with id d2c08238389cc3f9c359c57e146f0a92e564e56a73c174c8e69808341f9391da Oct 30 20:36:29.570131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.570037 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bt9s"] Oct 30 20:36:29.570916 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:29.570881 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdlwf"] Oct 30 20:36:30.157978 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:30.157942 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdlwf" event={"ID":"5aedd749-52bf-473f-86bb-f42e2cd459d0","Type":"ContainerStarted","Data":"44b85c7dd7a66f1761d678d6714adea2eda1b6740e6d239a62ca772c805ad279"} Oct 30 20:36:30.158459 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:30.157997 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdlwf" event={"ID":"5aedd749-52bf-473f-86bb-f42e2cd459d0","Type":"ContainerStarted","Data":"d2c08238389cc3f9c359c57e146f0a92e564e56a73c174c8e69808341f9391da"} Oct 30 20:36:30.159233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:30.159207 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bt9s" event={"ID":"b3970765-0603-41f1-9802-a252c08fdce7","Type":"ContainerStarted","Data":"e99fb4df29cd7056b23deb20b0025851e8d49c25c17fa085965b0b403ed5e31e"} Oct 30 20:36:30.160311 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:30.160277 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" event={"ID":"7ea62097-316d-49f5-bccf-dba48029bb6d","Type":"ContainerStarted","Data":"a26976f9d3eecd363b27e91e1d8e9bca4130ae9445d61f143c76b383981d7bec"} Oct 30 20:36:31.167453 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:31.167397 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdlwf" event={"ID":"5aedd749-52bf-473f-86bb-f42e2cd459d0","Type":"ContainerStarted","Data":"ca1b472102dd3c03d50cfd14a7c1d1226027120890c2f7475d37205680e28f84"} Oct 30 20:36:31.169465 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:31.169383 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bt9s" event={"ID":"b3970765-0603-41f1-9802-a252c08fdce7","Type":"ContainerStarted","Data":"37132ec399b4b28d89ca12656d7a24062c3977fe3001f6d1b5565b2b9833b820"} Oct 30 20:36:32.176631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:32.176587 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bt9s" event={"ID":"b3970765-0603-41f1-9802-a252c08fdce7","Type":"ContainerStarted","Data":"9884b353657bf2270c9664d0f7d5bb5192abab6a4d2cdc8610478d10b9be4aba"} Oct 30 20:36:32.178986 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:32.178956 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tvpjf" event={"ID":"7ea62097-316d-49f5-bccf-dba48029bb6d","Type":"ContainerStarted","Data":"396b0d162987c990f1a52129933fa903e288865227aa4a4e81ec3dbc1183dc63"} Oct 30 20:36:32.944615 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:32.944572 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:33.163276 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:33.163238 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:33.185908 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:33.185872 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdlwf" event={"ID":"5aedd749-52bf-473f-86bb-f42e2cd459d0","Type":"ContainerStarted","Data":"73ff3dc79dd898e4d2110ab5819221da4885514650a6c81aa7433c29b29d07da"} Oct 30 20:36:33.308924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:33.308895 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:33.355980 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:33.355941 4988 patch_prober.go:29] interesting pod/image-registry-587bd9b797-w6c54 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 20:36:33.356118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:33.356002 4988 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:36:33.904236 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:33.904203 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85df8d78bd-v8dx4"] Oct 30 20:36:34.060752 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:34.060718 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:34.139830 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:34.139801 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8jh5r" Oct 30 20:36:34.348020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:34.347935 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:34.562733 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:34.562695 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:34.816099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:34.816063 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:36.673509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:36.673472 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27nvb" Oct 30 20:36:37.044190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:37.044112 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:37.237748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:37.237710 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:37.256562 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:37.256530 4988 patch_prober.go:29] interesting pod/image-registry-587bd9b797-w6c54 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 20:36:37.256716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:37.256578 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:36:38.329108 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:38.329070 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:38.478836 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:38.478798 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:38.627909 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:38.627825 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:38.787108 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:38.787076 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:41.276670 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:41.276639 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:41.469919 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:41.469887 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:43.049590 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:43.049548 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:43.220089 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:43.220051 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:43.518684 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:43.518651 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:43.726162 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:43.726128 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:43.885897 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:43.885864 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:43.910952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:43.910921 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:44.038405 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.038370 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:44.202803 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.202726 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:44.293356 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.293324 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9zg62"] Oct 30 20:36:44.325055 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.325019 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.384188 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.384146 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:44.445698 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.445660 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:36:44.445880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.445759 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:36:44.470766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470679 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-tls\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.470766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470716 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp9qt\" (UniqueName: \"kubernetes.io/projected/4538254f-130c-4b0a-b8c4-2fc095d0758d-kube-api-access-vp9qt\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.470766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470735 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-sys\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.471008 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470878 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-root\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.471008 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470905 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.471008 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470939 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4538254f-130c-4b0a-b8c4-2fc095d0758d-metrics-client-ca\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.471008 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.470994 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.471178 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.471077 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-textfile\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.471178 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.471102 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-wtmp\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.494075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.494041 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5v6zk\"" Oct 30 20:36:44.494075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.494058 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:36:44.494075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.494049 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:36:44.539561 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.539524 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:44.571530 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571494 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-root\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571530 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571527 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571781 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571569 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4538254f-130c-4b0a-b8c4-2fc095d0758d-metrics-client-ca\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571781 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571591 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571781 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571594 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-root\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571781 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571664 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-textfile\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571781 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571714 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-wtmp\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.571781 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571767 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-tls\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572093 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571792 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp9qt\" (UniqueName: \"kubernetes.io/projected/4538254f-130c-4b0a-b8c4-2fc095d0758d-kube-api-access-vp9qt\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572093 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571816 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-sys\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572093 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571901 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-sys\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572093 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.571929 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-wtmp\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.572144 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.572220 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4538254f-130c-4b0a-b8c4-2fc095d0758d-metrics-client-ca\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.572527 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.572507 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-textfile\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.574061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.574037 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.574263 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.574245 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4538254f-130c-4b0a-b8c4-2fc095d0758d-node-exporter-tls\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.633240 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.633211 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp9qt\" (UniqueName: \"kubernetes.io/projected/4538254f-130c-4b0a-b8c4-2fc095d0758d-kube-api-access-vp9qt\") pod \"node-exporter-9zg62\" (UID: \"4538254f-130c-4b0a-b8c4-2fc095d0758d\") " pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.678199 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.678168 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9zg62" Oct 30 20:36:44.691435 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.691389 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:44.723241 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:36:44.723168 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4538254f_130c_4b0a_b8c4_2fc095d0758d.slice/crio-76998f83492c45a4358dcb649d76c83e03a053fca37c0bc6cb3d2834b38934f1 WatchSource:0}: Error finding container 76998f83492c45a4358dcb649d76c83e03a053fca37c0bc6cb3d2834b38934f1: Status 404 returned error can't find the container with id 76998f83492c45a4358dcb649d76c83e03a053fca37c0bc6cb3d2834b38934f1 Oct 30 20:36:44.850561 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.850514 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:44.997906 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:44.997816 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:45.236866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:45.236830 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zg62" event={"ID":"4538254f-130c-4b0a-b8c4-2fc095d0758d","Type":"ContainerStarted","Data":"76998f83492c45a4358dcb649d76c83e03a053fca37c0bc6cb3d2834b38934f1"} Oct 30 20:36:45.263569 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:45.263486 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:45.407472 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:45.407439 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:45.625826 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:45.625736 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:45.873867 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:45.873834 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:46.233947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:46.233913 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:46.243510 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:46.243482 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zg62" event={"ID":"4538254f-130c-4b0a-b8c4-2fc095d0758d","Type":"ContainerStarted","Data":"c380e89f9460b78877d7251dbdcacbe150562ecfa1463e2740d18cb83852dc2f"} Oct 30 20:36:46.416639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:46.416561 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:46.588326 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:46.588294 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:47.055178 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.055140 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:47.149103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.149073 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jdq2q" Oct 30 20:36:47.221582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.221551 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:47.248994 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.248962 4988 generic.go:356] "Generic (PLEG): container finished" podID="4538254f-130c-4b0a-b8c4-2fc095d0758d" containerID="c380e89f9460b78877d7251dbdcacbe150562ecfa1463e2740d18cb83852dc2f" exitCode=0 Oct 30 20:36:47.249376 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.249003 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zg62" event={"ID":"4538254f-130c-4b0a-b8c4-2fc095d0758d","Type":"ContainerDied","Data":"c380e89f9460b78877d7251dbdcacbe150562ecfa1463e2740d18cb83852dc2f"} Oct 30 20:36:47.255928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.255906 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:36:47.845114 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:47.845083 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:48.055145 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.055106 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:48.255068 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.255035 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zg62" event={"ID":"4538254f-130c-4b0a-b8c4-2fc095d0758d","Type":"ContainerStarted","Data":"6dcb60aa7ce79c34a7a48b11a767ef009dc24e8530a6787113e0b88a88cf91b0"} Oct 30 20:36:48.255483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.255085 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9zg62" event={"ID":"4538254f-130c-4b0a-b8c4-2fc095d0758d","Type":"ContainerStarted","Data":"4a84b5960569e058796e6eed6e545dc769f386c5bdb85912a08f0b9f9d421416"} Oct 30 20:36:48.335120 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.335090 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:48.391704 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.391638 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9zg62" podStartSLOduration=2.8957644030000003 podStartE2EDuration="4.391616432s" totalImagesPullingTime="1.426514156s" totalInitContainerRuntime="69.337873ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:48.390707847 +0000 UTC m=+36.498203758" watchObservedRunningTime="2026-10-30 20:36:48.391616432 +0000 UTC m=+36.499112341" Oct 30 20:36:48.485525 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.485479 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:48.685749 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.685713 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:48.836944 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.836900 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:48.984075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:48.983972 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:49.127669 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:49.127616 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:49.648705 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:49.648663 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:49.842926 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:49.842886 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:52.847239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:52.847157 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:53.045297 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:53.045264 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:53.257760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:53.257725 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:54.650009 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:54.649974 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:54.813459 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:54.813404 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:56.393929 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:56.393890 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:36:56.403754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:56.403726 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:56.559779 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:56.559743 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:56.715612 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:56.715539 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-587bd9b797-w6c54"] Oct 30 20:36:56.834113 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:56.834081 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:56.991869 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:56.991780 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:57.191054 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:57.191024 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:58.077844 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:58.077809 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:58.299139 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:58.299109 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:58.921236 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:58.921204 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:58.924547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:58.924518 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" podUID="be31e37c-2894-420c-84cc-16fa0776f92c" containerName="registry" containerID="cri-o://8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2" gracePeriod=30 Oct 30 20:36:59.084342 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.084302 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:59.165526 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.165500 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:59.194516 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194439 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194516 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194510 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194685 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194558 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194773 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194752 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194826 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194803 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194835 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194878 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.194932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194907 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") pod \"be31e37c-2894-420c-84cc-16fa0776f92c\" (UID: \"be31e37c-2894-420c-84cc-16fa0776f92c\") " Oct 30 20:36:59.195164 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.194904 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:59.195423 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.195385 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:59.197223 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.197193 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "kube-api-access-wdn8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:59.197326 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.197288 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:59.197479 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.197458 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:59.197548 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.197521 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:59.197604 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.197552 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:59.201387 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.201362 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted" pod "be31e37c-2894-420c-84cc-16fa0776f92c" (UID: "be31e37c-2894-420c-84cc-16fa0776f92c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:36:59.295640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295606 4988 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-registry-certificates\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295637 4988 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-bound-sa-token\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295648 4988 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be31e37c-2894-420c-84cc-16fa0776f92c-trusted-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295868 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295666 4988 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-image-registry-private-configuration\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295868 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295677 4988 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be31e37c-2894-420c-84cc-16fa0776f92c-ca-trust-extracted\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295868 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295685 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdn8m\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-kube-api-access-wdn8m\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295868 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295694 4988 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be31e37c-2894-420c-84cc-16fa0776f92c-installation-pull-secrets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.295868 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.295704 4988 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be31e37c-2894-420c-84cc-16fa0776f92c-registry-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:36:59.300136 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.300101 4988 generic.go:356] "Generic (PLEG): container finished" podID="be31e37c-2894-420c-84cc-16fa0776f92c" containerID="8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2" exitCode=0 Oct 30 20:36:59.300276 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.300144 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" Oct 30 20:36:59.300276 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.300148 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" event={"ID":"be31e37c-2894-420c-84cc-16fa0776f92c","Type":"ContainerDied","Data":"8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2"} Oct 30 20:36:59.300351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.300308 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85df8d78bd-v8dx4" event={"ID":"be31e37c-2894-420c-84cc-16fa0776f92c","Type":"ContainerDied","Data":"5b0495c216aafb871df12109301300115ad59e1a51fac18e3de2d747fa99cb50"} Oct 30 20:36:59.300351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.300338 4988 scope.go:122] "RemoveContainer" containerID="8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2" Oct 30 20:36:59.309881 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.309858 4988 scope.go:122] "RemoveContainer" containerID="8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2" Oct 30 20:36:59.310130 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:36:59.310108 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2\": container with ID starting with 8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2 not found: ID does not exist" containerID="8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2" Oct 30 20:36:59.310178 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.310139 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2"} err="failed to get container status \"8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2\": rpc error: code = NotFound desc = could not find container \"8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2\": container with ID starting with 8c53f34d415835e58df415877158ecd90745cfc47b25ab5196b3ffbed35c29a2 not found: ID does not exist" Oct 30 20:36:59.463358 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.463277 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85df8d78bd-v8dx4"] Oct 30 20:36:59.467233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.467204 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85df8d78bd-v8dx4"] Oct 30 20:36:59.581014 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.580981 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:59.739109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:36:59.739025 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:00.018159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:00.018088 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be31e37c-2894-420c-84cc-16fa0776f92c" path="/var/lib/kubelet/pods/be31e37c-2894-420c-84cc-16fa0776f92c/volumes" Oct 30 20:37:00.826222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:00.826188 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:00.986652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:00.986615 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:01.605154 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:01.605120 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:01.805564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:01.805531 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:06.797207 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:06.797172 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:06.950492 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:06.950455 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:14.577784 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:14.577743 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:14.765538 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:14.765481 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:18.654612 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:18.654573 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:21.735026 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:21.734984 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerName="registry" containerID="cri-o://07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2" gracePeriod=30 Oct 30 20:37:21.984361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:21.984335 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:37:22.085794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085720 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.085794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085765 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.085973 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085804 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.085973 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085842 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.085973 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085898 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.085973 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085924 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.086168 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.085971 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.086168 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.086001 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") pod \"391e9838-62c0-4d2d-8d8e-29d7493eff20\" (UID: \"391e9838-62c0-4d2d-8d8e-29d7493eff20\") " Oct 30 20:37:22.086371 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.086272 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:22.086644 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.086613 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:22.088227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.088170 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:22.088748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.088716 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:22.088849 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.088825 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:22.088903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.088844 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "kube-api-access-vtvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:22.088943 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.088888 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:22.093017 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.092980 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted" pod "391e9838-62c0-4d2d-8d8e-29d7493eff20" (UID: "391e9838-62c0-4d2d-8d8e-29d7493eff20"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:22.187452 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187401 4988 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-certificates\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187452 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187449 4988 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/391e9838-62c0-4d2d-8d8e-29d7493eff20-ca-trust-extracted\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187452 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187461 4988 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-image-registry-private-configuration\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187691 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187473 4988 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-bound-sa-token\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187691 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187483 4988 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/391e9838-62c0-4d2d-8d8e-29d7493eff20-installation-pull-secrets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187691 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187492 4988 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/391e9838-62c0-4d2d-8d8e-29d7493eff20-trusted-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187691 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187500 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvtw\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-kube-api-access-vtvtw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.187691 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.187509 4988 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/391e9838-62c0-4d2d-8d8e-29d7493eff20-registry-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:22.391114 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.391081 4988 generic.go:356] "Generic (PLEG): container finished" podID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerID="07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2" exitCode=0 Oct 30 20:37:22.391373 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.391125 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" Oct 30 20:37:22.391373 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.391178 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" event={"ID":"391e9838-62c0-4d2d-8d8e-29d7493eff20","Type":"ContainerDied","Data":"07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2"} Oct 30 20:37:22.391373 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.391221 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587bd9b797-w6c54" event={"ID":"391e9838-62c0-4d2d-8d8e-29d7493eff20","Type":"ContainerDied","Data":"740132b84afa0343c5794b9b4fa7fde34a7feff826d34f68d4a37cb01f772ce5"} Oct 30 20:37:22.391373 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.391239 4988 scope.go:122] "RemoveContainer" containerID="07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2" Oct 30 20:37:22.402080 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.402055 4988 scope.go:122] "RemoveContainer" containerID="07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2" Oct 30 20:37:22.402450 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:37:22.402399 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2\": container with ID starting with 07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2 not found: ID does not exist" containerID="07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2" Oct 30 20:37:22.402579 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.402461 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2"} err="failed to get container status \"07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2\": rpc error: code = NotFound desc = could not find container \"07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2\": container with ID starting with 07ea6244f19f899aa48e7ee38d94ec27aa6fc38e7f67b1aa3971a64f7a1bb2d2 not found: ID does not exist" Oct 30 20:37:22.554988 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.554954 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-587bd9b797-w6c54"] Oct 30 20:37:22.559337 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:22.559312 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-587bd9b797-w6c54"] Oct 30 20:37:24.014622 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:24.014590 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" path="/var/lib/kubelet/pods/391e9838-62c0-4d2d-8d8e-29d7493eff20/volumes" Oct 30 20:37:26.393548 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:37:26.393518 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:37:29.564151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:29.564121 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:29.733356 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:29.733322 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:33.434743 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:33.434710 4988 generic.go:356] "Generic (PLEG): container finished" podID="0a3af4a4-5731-41d0-8f14-cbe1543de1f7" containerID="9060c5c33ce9c0d35afc45658825b75fc2a44cb53eca60df3681ee64418676c4" exitCode=0 Oct 30 20:37:33.435138 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:33.434790 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" event={"ID":"0a3af4a4-5731-41d0-8f14-cbe1543de1f7","Type":"ContainerDied","Data":"9060c5c33ce9c0d35afc45658825b75fc2a44cb53eca60df3681ee64418676c4"} Oct 30 20:37:33.435191 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:33.435171 4988 scope.go:122] "RemoveContainer" containerID="9060c5c33ce9c0d35afc45658825b75fc2a44cb53eca60df3681ee64418676c4" Oct 30 20:37:34.444251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:34.444217 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-88htp" event={"ID":"0a3af4a4-5731-41d0-8f14-cbe1543de1f7","Type":"ContainerStarted","Data":"5e9b36342410fe9d67328960c728c3fd23c74b81c01252fb34967881f6c27a71"} Oct 30 20:37:38.460989 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:38.460952 4988 generic.go:356] "Generic (PLEG): container finished" podID="c365d189-b04a-4b12-8519-30c074796180" containerID="e0e26d7453e756e3ec10d76eca29ed2b2b51047c590ebcca688a17be73d87b77" exitCode=0 Oct 30 20:37:38.461358 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:38.461025 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" event={"ID":"c365d189-b04a-4b12-8519-30c074796180","Type":"ContainerDied","Data":"e0e26d7453e756e3ec10d76eca29ed2b2b51047c590ebcca688a17be73d87b77"} Oct 30 20:37:38.461358 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:38.461355 4988 scope.go:122] "RemoveContainer" containerID="e0e26d7453e756e3ec10d76eca29ed2b2b51047c590ebcca688a17be73d87b77" Oct 30 20:37:39.466636 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:39.466605 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tmxhb" event={"ID":"c365d189-b04a-4b12-8519-30c074796180","Type":"ContainerStarted","Data":"10e110a2147a390f18cf5b8c84ccfe3aa03bfc5d4b20f30b22f1f08b29936e99"} Oct 30 20:37:39.747196 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:39.747118 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jfr7m-mw5r6"] Oct 30 20:37:39.747338 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:39.747297 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" podUID="99eaa580-8ff6-4a47-971b-80133022ccf0" containerName="insights-gathering" containerID="cri-o://6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690" gracePeriod=30 Oct 30 20:37:39.982866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:39.982837 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:37:40.132621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.132593 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") pod \"99eaa580-8ff6-4a47-971b-80133022ccf0\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " Oct 30 20:37:40.132777 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.132641 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") pod \"99eaa580-8ff6-4a47-971b-80133022ccf0\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " Oct 30 20:37:40.132777 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.132670 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") pod \"99eaa580-8ff6-4a47-971b-80133022ccf0\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " Oct 30 20:37:40.132777 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.132703 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") pod \"99eaa580-8ff6-4a47-971b-80133022ccf0\" (UID: \"99eaa580-8ff6-4a47-971b-80133022ccf0\") " Oct 30 20:37:40.132975 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.132951 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path" pod "99eaa580-8ff6-4a47-971b-80133022ccf0" (UID: "99eaa580-8ff6-4a47-971b-80133022ccf0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:37:40.133070 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.132992 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle" pod "99eaa580-8ff6-4a47-971b-80133022ccf0" (UID: "99eaa580-8ff6-4a47-971b-80133022ccf0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:40.133203 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.133177 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle" pod "99eaa580-8ff6-4a47-971b-80133022ccf0" (UID: "99eaa580-8ff6-4a47-971b-80133022ccf0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:40.134975 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.134951 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c" pod "99eaa580-8ff6-4a47-971b-80133022ccf0" (UID: "99eaa580-8ff6-4a47-971b-80133022ccf0"). InnerVolumeSpecName "kube-api-access-xbx6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:40.233530 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.233497 4988 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99eaa580-8ff6-4a47-971b-80133022ccf0-archives-path\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:40.233530 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.233524 4988 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-service-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:40.233530 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.233537 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbx6c\" (UniqueName: \"kubernetes.io/projected/99eaa580-8ff6-4a47-971b-80133022ccf0-kube-api-access-xbx6c\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:40.233759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.233546 4988 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99eaa580-8ff6-4a47-971b-80133022ccf0-trusted-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:37:40.475905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.475813 4988 generic.go:356] "Generic (PLEG): container finished" podID="99eaa580-8ff6-4a47-971b-80133022ccf0" containerID="6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690" exitCode=2 Oct 30 20:37:40.475905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.475877 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" Oct 30 20:37:40.475905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.475890 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" event={"ID":"99eaa580-8ff6-4a47-971b-80133022ccf0","Type":"ContainerDied","Data":"6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690"} Oct 30 20:37:40.476396 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.475974 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jfr7m-mw5r6" event={"ID":"99eaa580-8ff6-4a47-971b-80133022ccf0","Type":"ContainerDied","Data":"6142043c46f4f685a4f45e3cebe44a8d5922e0c21f46ef672f7920d86dd5ee08"} Oct 30 20:37:40.476396 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.476001 4988 scope.go:122] "RemoveContainer" containerID="6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690" Oct 30 20:37:40.483873 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.483856 4988 scope.go:122] "RemoveContainer" containerID="6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690" Oct 30 20:37:40.484122 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:37:40.484104 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690\": container with ID starting with 6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690 not found: ID does not exist" containerID="6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690" Oct 30 20:37:40.484290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.484136 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690"} err="failed to get container status \"6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690\": rpc error: code = NotFound desc = could not find container \"6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690\": container with ID starting with 6c78230d4c6780bced0f07e4f525cb61fb1650f7e31c2ae6bbb2f73de9e32690 not found: ID does not exist" Oct 30 20:37:40.641006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.640969 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jfr7m-mw5r6"] Oct 30 20:37:40.690204 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:40.690169 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jfr7m-mw5r6"] Oct 30 20:37:42.014572 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:42.014542 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99eaa580-8ff6-4a47-971b-80133022ccf0" path="/var/lib/kubelet/pods/99eaa580-8ff6-4a47-971b-80133022ccf0/volumes" Oct 30 20:37:42.484685 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:42.484649 4988 generic.go:356] "Generic (PLEG): container finished" podID="cd681674-072e-4dff-a369-9d956c32b960" containerID="315e65897483b7b8d176bf98cfd0e9b88a3e310223d32046e39511f5308c4211" exitCode=0 Oct 30 20:37:42.484854 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:42.484722 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" event={"ID":"cd681674-072e-4dff-a369-9d956c32b960","Type":"ContainerDied","Data":"315e65897483b7b8d176bf98cfd0e9b88a3e310223d32046e39511f5308c4211"} Oct 30 20:37:42.485053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:42.485031 4988 scope.go:122] "RemoveContainer" containerID="315e65897483b7b8d176bf98cfd0e9b88a3e310223d32046e39511f5308c4211" Oct 30 20:37:43.490634 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:37:43.490598 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-h6tkv" event={"ID":"cd681674-072e-4dff-a369-9d956c32b960","Type":"ContainerStarted","Data":"363a231a04ee7235b670b18395667d083a06f31ebaab573b2a478ec55aafd58c"} Oct 30 20:37:56.393739 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:37:56.393703 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:38:01.926609 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:01.926568 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:13.018229 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:13.018192 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:13.222872 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:13.222819 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:15.683757 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:15.683719 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:15.879672 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:15.879622 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:16.946155 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:16.946116 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:17.149807 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:17.149768 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:25.675285 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:25.675190 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:25.821292 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:25.821239 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:26.394026 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:38:26.393983 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:38:29.561734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:29.561690 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:29.711832 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:29.711794 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:31.657093 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:31.657054 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:31.802111 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:38:31.802045 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.392871 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:38:56.392835 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:39:02.273125 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:39:02.273071 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:26.392747 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:39:26.392707 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:39:56.393553 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:39:56.393475 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:40:02.613143 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:40:02.613111 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:21.554884 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:40:21.554842 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:22.410484 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:40:22.410449 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:26.393707 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:40:26.393669 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:40:39.639067 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:40:39.639031 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:56.395178 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:40:56.395139 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:41:02.969933 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:41:02.969896 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:03.610286 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:41:03.610243 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:06.550233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:41:06.550195 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:06.744170 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:41:06.744119 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:12.028860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:41:12.028832 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:41:12.029546 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:41:12.029529 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:41:26.393400 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:41:26.393320 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:41:56.393758 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:41:56.393719 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:42:03.461475 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:42:03.461433 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:26.392217 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:42:26.392178 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:42:45.401907 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:42:45.401868 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:45.580815 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:42:45.580777 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:56.393660 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:42:56.393585 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:43:03.648201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:03.648164 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:12.959182 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:12.959141 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:13.890328 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:13.890286 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:16.350277 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:16.350234 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:16.614508 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:16.614401 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:20.313571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:20.313520 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:20.492994 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:20.492958 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:21.564395 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:21.564354 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:21.787577 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:21.787545 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:22.738101 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:22.738038 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:23.127998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:23.127884 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:26.393380 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:43:26.393345 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:43:30.907117 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:30.907058 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:31.113078 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:31.113043 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:40.933970 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.933935 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-bdtzw"] Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934225 4988 cpu_manager.go:414] "containerMap: removing container" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerName="registry" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934236 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerName="registry" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934259 4988 cpu_manager.go:414] "containerMap: removing container" podUID="be31e37c-2894-420c-84cc-16fa0776f92c" containerName="registry" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934265 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be31e37c-2894-420c-84cc-16fa0776f92c" containerName="registry" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934274 4988 cpu_manager.go:414] "containerMap: removing container" podUID="99eaa580-8ff6-4a47-971b-80133022ccf0" containerName="insights-gathering" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934278 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99eaa580-8ff6-4a47-971b-80133022ccf0" containerName="insights-gathering" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934346 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="be31e37c-2894-420c-84cc-16fa0776f92c" containerName="registry" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934355 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="99eaa580-8ff6-4a47-971b-80133022ccf0" containerName="insights-gathering" Oct 30 20:43:40.934363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.934362 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="391e9838-62c0-4d2d-8d8e-29d7493eff20" containerName="registry" Oct 30 20:43:40.937166 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.937150 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:40.946153 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.946128 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklr7\" (UniqueName: \"kubernetes.io/projected/7b08051b-dd7a-49f8-9805-afbca0971a23-kube-api-access-zklr7\") pod \"cert-manager-597b95cb84-bdtzw\" (UID: \"7b08051b-dd7a-49f8-9805-afbca0971a23\") " pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:40.946266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.946215 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b08051b-dd7a-49f8-9805-afbca0971a23-bound-sa-token\") pod \"cert-manager-597b95cb84-bdtzw\" (UID: \"7b08051b-dd7a-49f8-9805-afbca0971a23\") " pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:40.990362 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:40.990326 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-g2dg4\"" Oct 30 20:43:41.040045 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.040014 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 20:43:41.040248 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.040183 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 20:43:41.041053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.041033 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bdtzw"] Oct 30 20:43:41.047586 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.047562 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b08051b-dd7a-49f8-9805-afbca0971a23-bound-sa-token\") pod \"cert-manager-597b95cb84-bdtzw\" (UID: \"7b08051b-dd7a-49f8-9805-afbca0971a23\") " pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:41.047759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.047729 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklr7\" (UniqueName: \"kubernetes.io/projected/7b08051b-dd7a-49f8-9805-afbca0971a23-kube-api-access-zklr7\") pod \"cert-manager-597b95cb84-bdtzw\" (UID: \"7b08051b-dd7a-49f8-9805-afbca0971a23\") " pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:41.108855 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.108819 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b08051b-dd7a-49f8-9805-afbca0971a23-bound-sa-token\") pod \"cert-manager-597b95cb84-bdtzw\" (UID: \"7b08051b-dd7a-49f8-9805-afbca0971a23\") " pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:41.109000 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.108915 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklr7\" (UniqueName: \"kubernetes.io/projected/7b08051b-dd7a-49f8-9805-afbca0971a23-kube-api-access-zklr7\") pod \"cert-manager-597b95cb84-bdtzw\" (UID: \"7b08051b-dd7a-49f8-9805-afbca0971a23\") " pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:41.261594 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.261504 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-bdtzw" Oct 30 20:43:41.632966 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.632938 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-bdtzw"] Oct 30 20:43:41.636975 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:43:41.636943 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b08051b_dd7a_49f8_9805_afbca0971a23.slice/crio-4912ad271ae02432cda5e3d8b7325bbce4053be02605d6c5e11d67d1697d0223 WatchSource:0}: Error finding container 4912ad271ae02432cda5e3d8b7325bbce4053be02605d6c5e11d67d1697d0223: Status 404 returned error can't find the container with id 4912ad271ae02432cda5e3d8b7325bbce4053be02605d6c5e11d67d1697d0223 Oct 30 20:43:41.638955 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.638937 4988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:43:41.769437 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:41.769386 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bdtzw" event={"ID":"7b08051b-dd7a-49f8-9805-afbca0971a23","Type":"ContainerStarted","Data":"4912ad271ae02432cda5e3d8b7325bbce4053be02605d6c5e11d67d1697d0223"} Oct 30 20:43:46.792445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:46.792369 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-bdtzw" event={"ID":"7b08051b-dd7a-49f8-9805-afbca0971a23","Type":"ContainerStarted","Data":"8d73f0b8b081fd84d048a2139c5c33f08affd685e1b843598f43a22b5fbea0de"} Oct 30 20:43:51.252277 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:51.252245 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:51.447375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:51.447327 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:51.738638 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:51.738602 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:51.887487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:51.887457 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:52.052131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:52.052037 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:52.195140 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:52.195097 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:53.464463 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:43:53.464424 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:56.393787 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:43:56.393754 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:44:04.090471 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:44:04.090431 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:26.393592 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:44:26.393509 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:44:45.246623 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:44:45.246584 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:45.391488 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:44:45.391439 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:53.816481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:44:53.816439 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.392501 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:44:56.392462 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:45:04.359553 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:04.359515 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:19.022140 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:19.022107 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:19.315914 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:19.315831 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:26.393713 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:45:26.393675 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:45:32.351039 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:32.350999 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:33.271615 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:33.271582 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:33.945251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:33.945216 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:35.525330 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:35.525295 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:35.747892 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:35.747857 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:36.167957 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:36.167923 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:36.376045 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:36.376008 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:36.605116 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:36.605033 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:36.855533 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:36.855444 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:37.040953 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:37.040907 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:37.713954 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:37.713921 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:38.081791 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:38.081695 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:38.371706 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:38.371617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:39.764941 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:39.764822 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.001258 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:40.001228 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:40.562325 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:40.562288 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:41.356667 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:41.356632 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:42.200646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:42.200608 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:42.422928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:42.422889 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:43.281130 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:43.281095 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:43.579566 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:43.579481 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:44.608952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:44.608915 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:44.839843 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:44.839806 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:46.133090 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:46.133057 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:46.373881 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:46.373844 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:47.639207 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:47.639169 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:47.881960 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:47.881926 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.091752 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:48.091652 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.277859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:48.277819 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.589015 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:48.588984 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:50.219195 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.219130 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-bdtzw" podStartSLOduration=125.497550381 podStartE2EDuration="2m10.219109347s" totalImagesPullingTime="4.721558966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:46.910372917 +0000 UTC m=+455.017868828" watchObservedRunningTime="2026-10-30 20:45:50.219109347 +0000 UTC m=+578.326605263" Oct 30 20:45:50.219847 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.219828 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5"] Oct 30 20:45:50.222915 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.222901 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.275674 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.275642 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-dockercfg-qnnrv\"" Oct 30 20:45:50.276462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.276450 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-service-cert\"" Oct 30 20:45:50.276638 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.276619 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:45:50.276677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.276662 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 30 20:45:50.327272 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.327240 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5"] Oct 30 20:45:50.343610 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.343577 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-apiservice-cert\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.343744 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.343633 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnn4\" (UniqueName: \"kubernetes.io/projected/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-kube-api-access-fdnn4\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.343744 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.343728 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-webhook-cert\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.343821 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.343775 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-openshift-service-ca\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.444433 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.444379 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-webhook-cert\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.444632 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.444453 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-openshift-service-ca\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.444632 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.444483 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-apiservice-cert\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.444632 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.444520 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnn4\" (UniqueName: \"kubernetes.io/projected/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-kube-api-access-fdnn4\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.445201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.445177 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-openshift-service-ca\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.446872 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.446853 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-apiservice-cert\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.446964 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.446944 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-webhook-cert\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.506551 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.506483 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnn4\" (UniqueName: \"kubernetes.io/projected/dd5d4a45-4f01-4b5c-9b23-bb3242348a22-kube-api-access-fdnn4\") pod \"perses-operator-5df9476f48-72wq5\" (UID: \"dd5d4a45-4f01-4b5c-9b23-bb3242348a22\") " pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:50.535085 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:50.535055 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:51.042047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:51.042007 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5"] Oct 30 20:45:51.064316 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:45:51.064282 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5d4a45_4f01_4b5c_9b23_bb3242348a22.slice/crio-cdb45250af929fea79947eccb7436ad854e21b080a1348002ec730958d28f4de WatchSource:0}: Error finding container cdb45250af929fea79947eccb7436ad854e21b080a1348002ec730958d28f4de: Status 404 returned error can't find the container with id cdb45250af929fea79947eccb7436ad854e21b080a1348002ec730958d28f4de Oct 30 20:45:51.266909 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:51.266873 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" event={"ID":"dd5d4a45-4f01-4b5c-9b23-bb3242348a22","Type":"ContainerStarted","Data":"cdb45250af929fea79947eccb7436ad854e21b080a1348002ec730958d28f4de"} Oct 30 20:45:52.674199 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:52.674107 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:53.505060 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:53.505027 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:53.799562 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:53.799465 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:53.975174 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:53.975000 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:54.193759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.193724 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:54.501637 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.501549 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:54.602339 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.602304 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx"] Oct 30 20:45:54.605675 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.605654 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.680467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.680434 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:45:54.680467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.680440 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 30 20:45:54.680693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.680488 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 30 20:45:54.685917 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.685884 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgvf\" (UniqueName: \"kubernetes.io/projected/8888cff1-cc18-4bfc-a82c-d4150294a781-kube-api-access-gtgvf\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.686077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.685969 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8888cff1-cc18-4bfc-a82c-d4150294a781-webhook-cert\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.686077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.686019 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8888cff1-cc18-4bfc-a82c-d4150294a781-apiservice-cert\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.722584 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.722550 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-747cn\"" Oct 30 20:45:54.736191 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.736028 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx"] Oct 30 20:45:54.737034 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.736958 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:54.787298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.787227 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8888cff1-cc18-4bfc-a82c-d4150294a781-webhook-cert\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.787298 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.787268 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8888cff1-cc18-4bfc-a82c-d4150294a781-apiservice-cert\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.787502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.787317 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgvf\" (UniqueName: \"kubernetes.io/projected/8888cff1-cc18-4bfc-a82c-d4150294a781-kube-api-access-gtgvf\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.789953 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.789924 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8888cff1-cc18-4bfc-a82c-d4150294a781-apiservice-cert\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.790074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.789928 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8888cff1-cc18-4bfc-a82c-d4150294a781-webhook-cert\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.871610 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.871571 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgvf\" (UniqueName: \"kubernetes.io/projected/8888cff1-cc18-4bfc-a82c-d4150294a781-kube-api-access-gtgvf\") pod \"opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx\" (UID: \"8888cff1-cc18-4bfc-a82c-d4150294a781\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:54.918013 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:54.917976 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:55.361284 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:55.361244 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx"] Oct 30 20:45:55.367843 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:45:55.367805 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8888cff1_cc18_4bfc_a82c_d4150294a781.slice/crio-82376ca4ff26e6aadeee6ff188a352baf78851399f812a346b9a24abd5aea49d WatchSource:0}: Error finding container 82376ca4ff26e6aadeee6ff188a352baf78851399f812a346b9a24abd5aea49d: Status 404 returned error can't find the container with id 82376ca4ff26e6aadeee6ff188a352baf78851399f812a346b9a24abd5aea49d Oct 30 20:45:56.289473 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:56.289439 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" event={"ID":"8888cff1-cc18-4bfc-a82c-d4150294a781","Type":"ContainerStarted","Data":"82376ca4ff26e6aadeee6ff188a352baf78851399f812a346b9a24abd5aea49d"} Oct 30 20:45:56.291219 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:56.291184 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" event={"ID":"dd5d4a45-4f01-4b5c-9b23-bb3242348a22","Type":"ContainerStarted","Data":"5865fc3f48ba292fc25e9d76a412d565b9b91be9cce21ef43c55fd24752e2143"} Oct 30 20:45:56.291364 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:56.291313 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:45:56.394388 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:45:56.394338 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:45:56.510234 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:56.510175 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" podStartSLOduration=1.7528275739999999 podStartE2EDuration="6.510160366s" totalImagesPullingTime="4.757332792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:56.509300554 +0000 UTC m=+584.616796465" watchObservedRunningTime="2026-10-30 20:45:56.510160366 +0000 UTC m=+584.617656278" Oct 30 20:45:57.002234 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:57.002190 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:57.164494 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:57.164458 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:57.369091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:57.368956 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:57.578645 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:57.578615 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:58.301591 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.301549 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" event={"ID":"8888cff1-cc18-4bfc-a82c-d4150294a781","Type":"ContainerStarted","Data":"069c769512fd09a728ccd1ede8feb3012d9c2b43a56b3f15e2077e2d91f43db2"} Oct 30 20:45:58.301772 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.301682 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:45:58.410020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.409988 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:58.491747 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.491682 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" podStartSLOduration=1.766757535 podStartE2EDuration="4.491666596s" totalImagesPullingTime="2.724909061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:58.441629403 +0000 UTC m=+586.549125315" watchObservedRunningTime="2026-10-30 20:45:58.491666596 +0000 UTC m=+586.599162504" Oct 30 20:45:58.614468 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.614436 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:58.817640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.817555 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:58.979608 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:45:58.979565 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:04.713717 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:04.713679 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:07.298707 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:07.298679 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/perses-operator-5df9476f48-72wq5" Oct 30 20:46:09.309468 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:09.309403 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-5fc9f4c8f5-vvgrx" Oct 30 20:46:12.057871 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:12.057841 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:46:12.058327 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:12.058255 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:46:14.763652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:14.763614 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:14.912663 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:14.912626 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.178376 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:15.178343 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.340841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:15.340804 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.152241 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:16.152201 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.364193 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:16.364157 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.547591 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:16.547496 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:16.700122 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:16.700087 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:24.274160 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:24.274117 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:26.394138 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:46:26.394103 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:46:28.400266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.400233 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg"] Oct 30 20:46:28.403417 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.403393 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.461984 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.461949 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\"" Oct 30 20:46:28.464249 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.464216 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.464372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.464256 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdbw\" (UniqueName: \"kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw\") pod \"opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.470887 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.470859 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:28.565723 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.565685 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.565895 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.565732 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdbw\" (UniqueName: \"kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw\") pod \"opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.568153 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.568123 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.623660 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.623621 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg"] Oct 30 20:46:28.629165 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.629135 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdbw\" (UniqueName: \"kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw\") pod \"opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:28.691029 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.690940 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:28.716342 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:28.716307 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:29.202700 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:29.202667 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg"] Oct 30 20:46:29.237635 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:46:29.237604 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda958a7a6_a05f_42c5_a93c_961628a8e2db.slice/crio-7dcd646bb9ee519cac8e3a2f8af00da46056aaf1cbdd50b84ce48e47adb47921 WatchSource:0}: Error finding container 7dcd646bb9ee519cac8e3a2f8af00da46056aaf1cbdd50b84ce48e47adb47921: Status 404 returned error can't find the container with id 7dcd646bb9ee519cac8e3a2f8af00da46056aaf1cbdd50b84ce48e47adb47921 Oct 30 20:46:29.427723 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:29.427682 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" event={"ID":"a958a7a6-a05f-42c5-a93c-961628a8e2db","Type":"ContainerStarted","Data":"f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e"} Oct 30 20:46:29.428109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:29.427739 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" event={"ID":"a958a7a6-a05f-42c5-a93c-961628a8e2db","Type":"ContainerStarted","Data":"7dcd646bb9ee519cac8e3a2f8af00da46056aaf1cbdd50b84ce48e47adb47921"} Oct 30 20:46:29.428109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:29.427776 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:29.536109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:29.536017 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg"] Oct 30 20:46:29.559157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:29.559102 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" podStartSLOduration=1.559088922 podStartE2EDuration="1.559088922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:29.557760314 +0000 UTC m=+617.665256232" watchObservedRunningTime="2026-10-30 20:46:29.559088922 +0000 UTC m=+617.666584827" Oct 30 20:46:31.436357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.436315 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" podUID="a958a7a6-a05f-42c5-a93c-961628a8e2db" containerName="manager" containerID="cri-o://f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e" gracePeriod=10 Oct 30 20:46:31.679621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.679589 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:31.794255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.794170 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw\" (UniqueName: \"kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw\") pod \"a958a7a6-a05f-42c5-a93c-961628a8e2db\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " Oct 30 20:46:31.794255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.794225 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert\") pod \"a958a7a6-a05f-42c5-a93c-961628a8e2db\" (UID: \"a958a7a6-a05f-42c5-a93c-961628a8e2db\") " Oct 30 20:46:31.796383 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.796351 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw" pod "a958a7a6-a05f-42c5-a93c-961628a8e2db" (UID: "a958a7a6-a05f-42c5-a93c-961628a8e2db"). InnerVolumeSpecName "kube-api-access-nxdbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:31.796517 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.796390 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert" pod "a958a7a6-a05f-42c5-a93c-961628a8e2db" (UID: "a958a7a6-a05f-42c5-a93c-961628a8e2db"). InnerVolumeSpecName "opentelemetry-operator-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:31.895633 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.895592 4988 reconciler_common.go:299] "Volume detached for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a958a7a6-a05f-42c5-a93c-961628a8e2db-opentelemetry-operator-metrics-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:31.895633 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:31.895627 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdbw\" (UniqueName: \"kubernetes.io/projected/a958a7a6-a05f-42c5-a93c-961628a8e2db-kube-api-access-nxdbw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:46:32.440863 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.440824 4988 generic.go:356] "Generic (PLEG): container finished" podID="a958a7a6-a05f-42c5-a93c-961628a8e2db" containerID="f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e" exitCode=2 Oct 30 20:46:32.441331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.440868 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" event={"ID":"a958a7a6-a05f-42c5-a93c-961628a8e2db","Type":"ContainerDied","Data":"f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e"} Oct 30 20:46:32.441331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.440875 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" Oct 30 20:46:32.441331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.440916 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg" event={"ID":"a958a7a6-a05f-42c5-a93c-961628a8e2db","Type":"ContainerDied","Data":"7dcd646bb9ee519cac8e3a2f8af00da46056aaf1cbdd50b84ce48e47adb47921"} Oct 30 20:46:32.441331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.440943 4988 scope.go:122] "RemoveContainer" containerID="f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e" Oct 30 20:46:32.449099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.449073 4988 scope.go:122] "RemoveContainer" containerID="f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e" Oct 30 20:46:32.449362 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:46:32.449341 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e\": container with ID starting with f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e not found: ID does not exist" containerID="f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e" Oct 30 20:46:32.449449 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.449370 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e"} err="failed to get container status \"f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e\": rpc error: code = NotFound desc = could not find container \"f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e\": container with ID starting with f33a42ad0550db4e4312e9c951f35ebd39e634c5f02e7f4f6db0017d51f8380e not found: ID does not exist" Oct 30 20:46:32.656032 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.655996 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg"] Oct 30 20:46:32.705811 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:32.705723 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-c68c4bf9c-2x6mg"] Oct 30 20:46:34.014997 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:34.014955 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a958a7a6-a05f-42c5-a93c-961628a8e2db" path="/var/lib/kubelet/pods/a958a7a6-a05f-42c5-a93c-961628a8e2db/volumes" Oct 30 20:46:39.361478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:39.361435 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:39.512266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:39.512226 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:39.834584 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:39.834548 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:40.124765 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:40.124678 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:41.672809 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:41.672767 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:41.889208 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:41.889171 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:43.163380 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:43.163350 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:43.325680 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:43.325646 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:43.607643 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:43.607534 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:43.754481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:43.754439 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:44.217762 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:44.217723 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:44.366959 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:44.366921 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:46.205365 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:46.205318 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:46.437131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:46.437094 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:48.825670 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:48.825636 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:49.022932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:49.022899 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:51.032402 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:51.032365 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:51.259846 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:51.259811 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:51.819180 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:51.819134 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:52.073821 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:46:52.073737 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:56.394259 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:46:56.394223 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:47:01.569905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:01.569873 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:01.768651 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:01.768607 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:05.091820 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:05.091787 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:14.773380 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:14.773344 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:14.918528 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:14.918486 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:15.659961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:15.659918 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:15.894734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:15.894695 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:16.067902 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:16.067816 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:18.063136 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.063102 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8"] Oct 30 20:47:18.063687 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.063433 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a958a7a6-a05f-42c5-a93c-961628a8e2db" containerName="manager" Oct 30 20:47:18.063687 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.063445 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a958a7a6-a05f-42c5-a93c-961628a8e2db" containerName="manager" Oct 30 20:47:18.063687 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.063519 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a958a7a6-a05f-42c5-a93c-961628a8e2db" containerName="manager" Oct 30 20:47:18.067605 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.067587 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.074854 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.074827 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-2wbh8\" (UID: \"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.074947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.074863 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxkn\" (UniqueName: \"kubernetes.io/projected/12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6-kube-api-access-sdxkn\") pod \"cluster-logging-operator-5d6db4c85-2wbh8\" (UID: \"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.120227 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.120191 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 30 20:47:18.169753 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.169722 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 30 20:47:18.169753 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.169738 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"cluster-logging-operator-dockercfg-wj8kk\"" Oct 30 20:47:18.169972 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.169825 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"cluster-logging-operator-metrics-certs\"" Oct 30 20:47:18.170966 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.170947 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8"] Oct 30 20:47:18.176206 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.176187 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-2wbh8\" (UID: \"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.176285 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.176222 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxkn\" (UniqueName: \"kubernetes.io/projected/12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6-kube-api-access-sdxkn\") pod \"cluster-logging-operator-5d6db4c85-2wbh8\" (UID: \"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.178594 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.178571 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-2wbh8\" (UID: \"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.237828 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.237798 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxkn\" (UniqueName: \"kubernetes.io/projected/12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6-kube-api-access-sdxkn\") pod \"cluster-logging-operator-5d6db4c85-2wbh8\" (UID: \"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.378967 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.378882 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" Oct 30 20:47:18.805506 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:18.805466 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8"] Oct 30 20:47:18.809629 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:47:18.809600 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12fd44a2_86a0_4cff_b98b_f6f31e9bc0b6.slice/crio-6f9d30ca1205e12913bae75aa536afe6e3cb3d4563bc9c3de7ffa6e145f3ba85 WatchSource:0}: Error finding container 6f9d30ca1205e12913bae75aa536afe6e3cb3d4563bc9c3de7ffa6e145f3ba85: Status 404 returned error can't find the container with id 6f9d30ca1205e12913bae75aa536afe6e3cb3d4563bc9c3de7ffa6e145f3ba85 Oct 30 20:47:19.485025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:19.484986 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:19.628709 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:19.628669 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:19.635245 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:19.635217 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" event={"ID":"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6","Type":"ContainerStarted","Data":"6f9d30ca1205e12913bae75aa536afe6e3cb3d4563bc9c3de7ffa6e145f3ba85"} Oct 30 20:47:24.660212 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:24.660172 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" event={"ID":"12fd44a2-86a0-4cff-b98b-f6f31e9bc0b6","Type":"ContainerStarted","Data":"c60a744ff585578e323c05aa867236ac3a08ecb31334cc635c34bf41d926b8ca"} Oct 30 20:47:24.808295 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:24.808236 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/cluster-logging-operator-5d6db4c85-2wbh8" podStartSLOduration=1.038886629 podStartE2EDuration="6.808220043s" totalImagesPullingTime="5.769333414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:24.807708423 +0000 UTC m=+672.915204374" watchObservedRunningTime="2026-10-30 20:47:24.808220043 +0000 UTC m=+672.915715952" Oct 30 20:47:26.394400 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:47:26.394360 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:47:27.032124 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:27.032083 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:27.200024 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:27.199989 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:36.748754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:36.748712 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:36.907861 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:36.907829 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:44.955611 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:44.955567 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:45.101944 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:45.101898 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:45.267773 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:45.267676 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:45.476942 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:45.476906 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:47.630216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:47.630175 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:47.802526 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:47.802490 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:47.953038 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:47.952994 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:47.977686 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:47.977653 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:47:47.981376 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:47.981352 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.096953 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.096917 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:47:48.097162 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.097013 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:47:48.097162 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.097136 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-f7h42\"" Oct 30 20:47:48.098083 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.098059 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:47:48.102714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.102685 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:48.115790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.115764 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbfh\" (UniqueName: \"kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.116136 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.116103 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.116254 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.116151 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.116322 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.116270 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.146571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.146534 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:47:48.146571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.146546 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:47:48.146571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.146546 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 20:47:48.216994 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.216912 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.216994 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.216948 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.216994 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.216965 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.217263 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.217000 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbfh\" (UniqueName: \"kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.217728 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.217700 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.219594 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.219572 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.219684 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.219632 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.278259 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.278221 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbfh\" (UniqueName: \"kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh\") pod \"data-science-collector-collector-1\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.295104 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.295067 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:47:48.670377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.670345 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:47:48.674611 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:47:48.674583 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5e7035_b960_4e5c_a2a7_40e2923f7ea6.slice/crio-c9f828dbad576ef8adcaf2cd4e2a3cac819fac961667ae76449507042f798f67 WatchSource:0}: Error finding container c9f828dbad576ef8adcaf2cd4e2a3cac819fac961667ae76449507042f798f67: Status 404 returned error can't find the container with id c9f828dbad576ef8adcaf2cd4e2a3cac819fac961667ae76449507042f798f67 Oct 30 20:47:48.764253 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:48.764206 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6","Type":"ContainerStarted","Data":"c9f828dbad576ef8adcaf2cd4e2a3cac819fac961667ae76449507042f798f67"} Oct 30 20:47:49.273355 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:49.273311 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:49.418727 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:49.418685 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:49.658648 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:49.658611 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:50.485399 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:50.485363 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:50.632900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:50.632862 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:50.955678 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:50.955637 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:51.599166 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.599130 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:47:51.602806 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.602786 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.631671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.631631 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:51.655366 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.655334 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 20:47:51.655600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.655398 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:47:51.655600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.655399 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 20:47:51.704755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.704712 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:47:51.704755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.704744 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 20:47:51.705009 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.704840 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 20:47:51.705009 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.704912 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 20:47:51.705125 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.705011 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-dm2rm\"" Oct 30 20:47:51.706677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.706651 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:47:51.827828 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.827795 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:51.847732 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.847695 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.847894 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.847740 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498tj\" (UniqueName: \"kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.847894 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.847823 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.847894 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.847886 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.847995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.847920 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.848028 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.847997 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.848028 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.848016 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.848091 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.848050 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949032 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.948999 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949229 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949043 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949229 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949062 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949229 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949199 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949266 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949309 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498tj\" (UniqueName: \"kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949350 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.949583 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949398 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.950017 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.949992 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.950129 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.950024 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.950129 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.950023 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.950290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.950268 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.951603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.951577 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.951717 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.951687 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:51.951807 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:51.951788 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:52.011158 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:52.011116 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-498tj\" (UniqueName: \"kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:54.750034 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:54.749997 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:47:54.750334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:54.750062 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:47:55.798304 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:55.798265 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6","Type":"ContainerStarted","Data":"8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c"} Oct 30 20:47:55.916106 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:55.916049 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.843421032 podStartE2EDuration="8.91603588s" totalImagesPullingTime="6.072614848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:55.915961905 +0000 UTC m=+704.023457829" watchObservedRunningTime="2026-10-30 20:47:55.91603588 +0000 UTC m=+704.023531795" Oct 30 20:47:56.394963 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:47:56.394928 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:47:59.017195 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.017147 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.122204 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.122168 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.218218 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.218177 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.218404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.218350 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.270652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.270562 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.480952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.480920 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.544036 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.543957 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:47:59.721749 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:47:59.721705 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:00.269731 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:00.269693 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:48:00.275501 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:48:00.275467 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583bbacf_dc81_4ee7_a7fb_b55735c7d799.slice/crio-cdfe689317998202037894ab0cefe6231d5d268d8d688043c16d79a12d61f788 WatchSource:0}: Error finding container cdfe689317998202037894ab0cefe6231d5d268d8d688043c16d79a12d61f788: Status 404 returned error can't find the container with id cdfe689317998202037894ab0cefe6231d5d268d8d688043c16d79a12d61f788 Oct 30 20:48:00.824237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:00.824184 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerStarted","Data":"cdfe689317998202037894ab0cefe6231d5d268d8d688043c16d79a12d61f788"} Oct 30 20:48:04.844400 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:04.844364 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerStarted","Data":"50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04"} Oct 30 20:48:05.390989 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:05.390950 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:08.863337 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:08.863300 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerStarted","Data":"c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d"} Oct 30 20:48:12.883039 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:12.882992 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerStarted","Data":"425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af"} Oct 30 20:48:12.883507 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:12.883099 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:12.883507 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:12.883127 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:12.885734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:12.885715 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:12.891356 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:12.891335 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:13.887851 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:13.887810 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:14.065870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.065835 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:48:14.066120 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.066091 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" containerName="otc-container" containerID="cri-o://8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c" gracePeriod=30 Oct 30 20:48:14.078463 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.078393 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:14.312328 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.312304 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:14.453834 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.453747 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs\") pod \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " Oct 30 20:48:14.453834 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.453784 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh\" (UniqueName: \"kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh\") pod \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " Oct 30 20:48:14.454058 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.453857 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs\") pod \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " Oct 30 20:48:14.454058 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.453893 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal\") pod \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\" (UID: \"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6\") " Oct 30 20:48:14.454326 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.454290 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal" pod "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" (UID: "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:14.456650 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.456610 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs" pod "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" (UID: "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:14.456650 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.456638 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs" pod "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" (UID: "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:14.456860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.456736 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh" pod "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" (UID: "4f5e7035-b960-4e5c-a2a7-40e2923f7ea6"). InnerVolumeSpecName "kube-api-access-kfbfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:14.470709 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.470668 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:14.554751 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.554711 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:14.554751 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.554743 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbfh\" (UniqueName: \"kubernetes.io/projected/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-kube-api-access-kfbfh\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:14.554751 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.554756 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:14.554988 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.554767 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:14.778646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.778562 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:48:14.781384 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.781354 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:14.893066 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.893035 4988 generic.go:356] "Generic (PLEG): container finished" podID="4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" containerID="8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c" exitCode=0 Oct 30 20:48:14.893568 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.893081 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:14.893568 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.893120 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6","Type":"ContainerDied","Data":"8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c"} Oct 30 20:48:14.893568 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.893168 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4f5e7035-b960-4e5c-a2a7-40e2923f7ea6","Type":"ContainerDied","Data":"c9f828dbad576ef8adcaf2cd4e2a3cac819fac961667ae76449507042f798f67"} Oct 30 20:48:14.893568 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.893191 4988 scope.go:122] "RemoveContainer" containerID="8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c" Oct 30 20:48:14.893792 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.893607 4988 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-dm2rm\" not found" Oct 30 20:48:14.903183 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.903168 4988 scope.go:122] "RemoveContainer" containerID="8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c" Oct 30 20:48:14.903471 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:14.903450 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c\": container with ID starting with 8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c not found: ID does not exist" containerID="8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c" Oct 30 20:48:14.903523 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:14.903483 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c"} err="failed to get container status \"8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c\": rpc error: code = NotFound desc = could not find container \"8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c\": container with ID starting with 8d6fc7a4863f3497bac3f9eae4fb27b09e74eb36c5fe161989f06893b370a10c not found: ID does not exist" Oct 30 20:48:15.031075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.030978 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:15.057137 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.057105 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:48:15.061294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.061262 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:48:15.160132 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160093 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160168 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660150253 +0000 UTC m=+723.767646161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160170 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160093 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160210 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660199433 +0000 UTC m=+723.767695357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160251 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160207 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160323 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660297354 +0000 UTC m=+723.767793257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160329 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160348 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660337956 +0000 UTC m=+723.767833848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.160385 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160365 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 30 20:48:15.160771 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160365 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660356206 +0000 UTC m=+723.767852098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 30 20:48:15.160771 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160445 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660428857 +0000 UTC m=+723.767924747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.160771 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.160462 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:15.660455471 +0000 UTC m=+723.767951360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 30 20:48:15.207280 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.207233 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:15.666338 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666297 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.666338 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666328 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666370 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.666356248 +0000 UTC m=+724.773852175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666384 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.666377882 +0000 UTC m=+724.773873773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666388 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666429 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666470 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.66645318 +0000 UTC m=+724.773949070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666490 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.666481289 +0000 UTC m=+724.773977178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 30 20:48:15.666633 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666616 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 30 20:48:15.666933 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666616 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 30 20:48:15.666933 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666649 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.666638731 +0000 UTC m=+724.774134624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 30 20:48:15.666933 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666681 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:48:15.666933 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666687 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.666677373 +0000 UTC m=+724.774173263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 30 20:48:15.666933 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:15.666726 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:16.666712998 +0000 UTC m=+724.774208890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:48:15.898290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.898256 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo" containerID="cri-o://50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04" gracePeriod=30 Oct 30 20:48:15.898835 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.898301 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway-opa" containerID="cri-o://425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af" gracePeriod=30 Oct 30 20:48:15.898835 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:15.898300 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway" containerID="cri-o://c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d" gracePeriod=30 Oct 30 20:48:16.016147 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:16.016074 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" path="/var/lib/kubelet/pods/4f5e7035-b960-4e5c-a2a7-40e2923f7ea6/volumes" Oct 30 20:48:16.675760 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675717 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675778 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675788 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675808 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675740 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675722 4988 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675803 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675807 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675788291 +0000 UTC m=+726.783284185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675897 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675878958 +0000 UTC m=+726.783374852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675911 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675903785 +0000 UTC m=+726.783399675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675921 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675915741 +0000 UTC m=+726.783411631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675936 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675930272 +0000 UTC m=+726.783426162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675950 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675944962 +0000 UTC m=+726.783440851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 30 20:48:16.675972 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:16.675962 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants podName:583bbacf-dc81-4ee7-a7fb-b55735c7d799 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:18.675958323 +0000 UTC m=+726.783454212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 30 20:48:16.904334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:16.904297 4988 generic.go:356] "Generic (PLEG): container finished" podID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerID="425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af" exitCode=0 Oct 30 20:48:16.904334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:16.904324 4988 generic.go:356] "Generic (PLEG): container finished" podID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerID="50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04" exitCode=0 Oct 30 20:48:16.904787 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:16.904375 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerDied","Data":"425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af"} Oct 30 20:48:16.904787 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:16.904450 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerDied","Data":"50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04"} Oct 30 20:48:17.162614 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.162592 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:17.280085 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280010 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280085 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280042 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280085 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280070 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280098 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj\" (UniqueName: \"kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280148 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280278 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280549 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280358 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280549 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280462 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:17.280549 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280505 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280715 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280568 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:17.280715 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280573 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:17.280715 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280617 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls\") pod \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\" (UID: \"583bbacf-dc81-4ee7-a7fb-b55735c7d799\") " Oct 30 20:48:17.280715 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280704 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:17.280924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280866 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.280924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280887 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.280924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280904 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.280924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.280918 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.282518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.282493 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "kube-api-access-498tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:17.282621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.282493 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:17.282995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.282972 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:17.283061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.283042 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:17.284165 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.284146 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "583bbacf-dc81-4ee7-a7fb-b55735c7d799" (UID: "583bbacf-dc81-4ee7-a7fb-b55735c7d799"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:48:17.382255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.382217 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:48:17.382255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.382250 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.382255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.382262 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.382525 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.382271 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-498tj\" (UniqueName: \"kubernetes.io/projected/583bbacf-dc81-4ee7-a7fb-b55735c7d799-kube-api-access-498tj\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.382525 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.382281 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/583bbacf-dc81-4ee7-a7fb-b55735c7d799-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:17.391251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.391224 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:48:17.483573 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.483531 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 20:48:17.910399 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.910362 4988 generic.go:356] "Generic (PLEG): container finished" podID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerID="c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d" exitCode=0 Oct 30 20:48:17.910880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.910427 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerDied","Data":"c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d"} Oct 30 20:48:17.910880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.910441 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:48:17.910880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.910474 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"583bbacf-dc81-4ee7-a7fb-b55735c7d799","Type":"ContainerDied","Data":"cdfe689317998202037894ab0cefe6231d5d268d8d688043c16d79a12d61f788"} Oct 30 20:48:17.910880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.910497 4988 scope.go:122] "RemoveContainer" containerID="425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af" Oct 30 20:48:17.918918 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.918889 4988 scope.go:122] "RemoveContainer" containerID="c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d" Oct 30 20:48:17.926548 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.926526 4988 scope.go:122] "RemoveContainer" containerID="50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04" Oct 30 20:48:17.934012 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.933995 4988 scope.go:122] "RemoveContainer" containerID="425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af" Oct 30 20:48:17.934266 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:17.934246 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af\": container with ID starting with 425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af not found: ID does not exist" containerID="425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af" Oct 30 20:48:17.934333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.934274 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af"} err="failed to get container status \"425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af\": rpc error: code = NotFound desc = could not find container \"425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af\": container with ID starting with 425c15cfebe505c70a4949d55e47443b89496e8ce86bf30f179ee870e60880af not found: ID does not exist" Oct 30 20:48:17.934333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.934295 4988 scope.go:122] "RemoveContainer" containerID="c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d" Oct 30 20:48:17.934540 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:17.934526 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d\": container with ID starting with c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d not found: ID does not exist" containerID="c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d" Oct 30 20:48:17.934587 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.934545 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d"} err="failed to get container status \"c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d\": rpc error: code = NotFound desc = could not find container \"c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d\": container with ID starting with c470bc8a2d374acd09715c33271570ffa1b98988a944caca2cd9794b66afed5d not found: ID does not exist" Oct 30 20:48:17.934587 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.934558 4988 scope.go:122] "RemoveContainer" containerID="50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04" Oct 30 20:48:17.934764 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:17.934750 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04\": container with ID starting with 50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04 not found: ID does not exist" containerID="50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04" Oct 30 20:48:17.934800 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:17.934768 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04"} err="failed to get container status \"50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04\": rpc error: code = NotFound desc = could not find container \"50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04\": container with ID starting with 50bbd2c76700403171276faea8221cbabb00e280f7531c4a3035e4a992edcd04 not found: ID does not exist" Oct 30 20:48:18.126172 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:18.126138 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:48:18.176006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:18.175920 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:48:20.015677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:20.015647 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" path="/var/lib/kubelet/pods/583bbacf-dc81-4ee7-a7fb-b55735c7d799/volumes" Oct 30 20:48:23.602429 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:23.602382 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:23.752274 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:23.752216 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:23.902419 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:23.902339 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:24.047935 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:24.047899 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:26.392702 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:26.392667 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:48:27.054048 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054018 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 20:48:27.054316 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054305 4988 cpu_manager.go:414] "containerMap: removing container" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo" Oct 30 20:48:27.054361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054318 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo" Oct 30 20:48:27.054361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054333 4988 cpu_manager.go:414] "containerMap: removing container" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway-opa" Oct 30 20:48:27.054361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054340 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway-opa" Oct 30 20:48:27.054361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054349 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" containerName="otc-container" Oct 30 20:48:27.054361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054353 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" containerName="otc-container" Oct 30 20:48:27.054361 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054362 4988 cpu_manager.go:414] "containerMap: removing container" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway" Oct 30 20:48:27.054557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054367 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway" Oct 30 20:48:27.054557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054480 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo" Oct 30 20:48:27.054557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054490 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway-opa" Oct 30 20:48:27.054557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054495 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5e7035-b960-4e5c-a2a7-40e2923f7ea6" containerName="otc-container" Oct 30 20:48:27.054557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.054501 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="583bbacf-dc81-4ee7-a7fb-b55735c7d799" containerName="tempo-gateway" Oct 30 20:48:27.056154 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.056139 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.088389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.088352 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:27.109727 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.109701 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:48:27.109854 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.109829 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-d65nb\"" Oct 30 20:48:27.153073 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.153027 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.153234 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.153130 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.153234 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.153171 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjf5\" (UniqueName: \"kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.153351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.153238 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.153351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.153335 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.159455 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.159405 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:48:27.159627 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.159613 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:48:27.159699 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.159649 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:48:27.159699 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.159679 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:48:27.159785 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.159708 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-b5f8018a\"" Oct 30 20:48:27.161768 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.161749 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 20:48:27.253735 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.253695 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.253928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.253744 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.253928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.253781 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.253928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.253812 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjf5\" (UniqueName: \"kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.253928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.253863 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.253928 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:27.253909 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 20:48:27.254208 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:27.253983 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert podName:4e98841f-a699-4631-bbf1-30266bde95ce nodeName:}" failed. No retries permitted until 2026-10-30 20:48:27.753966417 +0000 UTC m=+735.861462307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce") : secret "data-science-collector-ta-client-cert" not found Oct 30 20:48:27.254468 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.254451 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.256212 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.256185 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.256311 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.256242 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.292739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.292711 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:27.315714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.315656 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjf5\" (UniqueName: \"kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.552042 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.552004 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:27.755036 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.754997 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:27.758014 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:27.757993 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:27.758122 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:27.758109 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 20:48:27.758168 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:27.758159 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert podName:4e98841f-a699-4631-bbf1-30266bde95ce nodeName:}" failed. No retries permitted until 2026-10-30 20:48:28.758146169 +0000 UTC m=+736.865642059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce") : secret "data-science-collector-ta-client-cert" not found Oct 30 20:48:28.765767 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:28.765715 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:28.766179 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:28.765891 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 20:48:28.766179 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:28.765964 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert podName:4e98841f-a699-4631-bbf1-30266bde95ce nodeName:}" failed. No retries permitted until 2026-10-30 20:48:30.765948684 +0000 UTC m=+738.873444574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce") : secret "data-science-collector-ta-client-cert" not found Oct 30 20:48:30.779640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:30.779597 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:30.782010 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:30.781975 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:30.967403 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:30.967373 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:31.343183 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:31.343147 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 20:48:31.375038 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:48:31.374998 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e98841f_a699_4631_bbf1_30266bde95ce.slice/crio-97cdfea94117072adfe5c0268251d3548581a5ce46339ffd1bb1237ae1c92ba6 WatchSource:0}: Error finding container 97cdfea94117072adfe5c0268251d3548581a5ce46339ffd1bb1237ae1c92ba6: Status 404 returned error can't find the container with id 97cdfea94117072adfe5c0268251d3548581a5ce46339ffd1bb1237ae1c92ba6 Oct 30 20:48:31.377193 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:31.377152 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:48:31.377261 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:31.377240 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:48:31.969689 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:31.969651 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4e98841f-a699-4631-bbf1-30266bde95ce","Type":"ContainerStarted","Data":"e1dcb3347c22db26660f251b19b5deebea9d8a75997d68204509f324a6fa1a29"} Oct 30 20:48:31.970086 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:31.969704 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4e98841f-a699-4631-bbf1-30266bde95ce","Type":"ContainerStarted","Data":"97cdfea94117072adfe5c0268251d3548581a5ce46339ffd1bb1237ae1c92ba6"} Oct 30 20:48:32.088151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:32.088098 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=5.088083117 podStartE2EDuration="5.088083117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:32.087548167 +0000 UTC m=+740.195044061" watchObservedRunningTime="2026-10-30 20:48:32.088083117 +0000 UTC m=+740.195579029" Oct 30 20:48:44.753696 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:44.753640 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:44.903312 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:44.903279 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:50.849876 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:50.849843 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 20:48:50.850441 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:50.850028 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4e98841f-a699-4631-bbf1-30266bde95ce" containerName="otc-container" containerID="cri-o://e1dcb3347c22db26660f251b19b5deebea9d8a75997d68204509f324a6fa1a29" gracePeriod=30 Oct 30 20:48:50.878061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:50.878025 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:51.060353 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.060315 4988 generic.go:356] "Generic (PLEG): container finished" podID="4e98841f-a699-4631-bbf1-30266bde95ce" containerID="e1dcb3347c22db26660f251b19b5deebea9d8a75997d68204509f324a6fa1a29" exitCode=0 Oct 30 20:48:51.060527 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.060370 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4e98841f-a699-4631-bbf1-30266bde95ce","Type":"ContainerDied","Data":"e1dcb3347c22db26660f251b19b5deebea9d8a75997d68204509f324a6fa1a29"} Oct 30 20:48:51.072291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.072231 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:51.092591 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.092567 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:51.138785 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.138752 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs\") pod \"4e98841f-a699-4631-bbf1-30266bde95ce\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " Oct 30 20:48:51.138947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.138797 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5\" (UniqueName: \"kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5\") pod \"4e98841f-a699-4631-bbf1-30266bde95ce\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " Oct 30 20:48:51.138947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.138828 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal\") pod \"4e98841f-a699-4631-bbf1-30266bde95ce\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " Oct 30 20:48:51.138947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.138868 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") pod \"4e98841f-a699-4631-bbf1-30266bde95ce\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " Oct 30 20:48:51.138947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.138893 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs\") pod \"4e98841f-a699-4631-bbf1-30266bde95ce\" (UID: \"4e98841f-a699-4631-bbf1-30266bde95ce\") " Oct 30 20:48:51.139190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.139165 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal" pod "4e98841f-a699-4631-bbf1-30266bde95ce" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:51.141133 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.141101 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs" pod "4e98841f-a699-4631-bbf1-30266bde95ce" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:51.141251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.141106 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5" pod "4e98841f-a699-4631-bbf1-30266bde95ce" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce"). InnerVolumeSpecName "kube-api-access-nzjf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:51.141251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.141142 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs" pod "4e98841f-a699-4631-bbf1-30266bde95ce" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:51.141251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.141157 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert" pod "4e98841f-a699-4631-bbf1-30266bde95ce" (UID: "4e98841f-a699-4631-bbf1-30266bde95ce"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:51.239542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.239501 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:51.239542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.239531 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:51.239542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.239547 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e98841f-a699-4631-bbf1-30266bde95ce-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:51.239830 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.239556 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjf5\" (UniqueName: \"kubernetes.io/projected/4e98841f-a699-4631-bbf1-30266bde95ce-kube-api-access-nzjf5\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:51.239830 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.239565 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4e98841f-a699-4631-bbf1-30266bde95ce-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:48:51.671071 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.671031 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:51.867025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:51.866983 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:52.065814 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:52.065786 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 20:48:52.065975 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:52.065818 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4e98841f-a699-4631-bbf1-30266bde95ce","Type":"ContainerDied","Data":"97cdfea94117072adfe5c0268251d3548581a5ce46339ffd1bb1237ae1c92ba6"} Oct 30 20:48:52.065975 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:52.065870 4988 scope.go:122] "RemoveContainer" containerID="e1dcb3347c22db26660f251b19b5deebea9d8a75997d68204509f324a6fa1a29" Oct 30 20:48:52.229454 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:52.229402 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 20:48:52.279133 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:52.279046 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 20:48:54.015157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:54.015124 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e98841f-a699-4631-bbf1-30266bde95ce" path="/var/lib/kubelet/pods/4e98841f-a699-4631-bbf1-30266bde95ce/volumes" Oct 30 20:48:56.393381 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:56.393331 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:48:56.911190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:56.911151 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:57.058036 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:57.057993 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:57.251859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:57.251773 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:57.394956 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:57.394920 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:58.834639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.834601 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:58.856564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.856533 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:48:58.856860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.856848 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4e98841f-a699-4631-bbf1-30266bde95ce" containerName="otc-container" Oct 30 20:48:58.856901 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.856861 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e98841f-a699-4631-bbf1-30266bde95ce" containerName="otc-container" Oct 30 20:48:58.856953 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.856944 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e98841f-a699-4631-bbf1-30266bde95ce" containerName="otc-container" Oct 30 20:48:58.859875 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.859859 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:58.906570 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.906539 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:58.906754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.906576 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5sd\" (UniqueName: \"kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:58.906754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.906613 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:58.906871 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.906766 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:58.919182 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.919150 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:48:58.946152 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.946115 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:48:58.961557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.961529 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:48:58.961712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.961697 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:48:58.961766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.961711 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-mkvcc\"" Oct 30 20:48:58.961766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.961712 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:48:58.967084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.967069 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-c631c892\"" Oct 30 20:48:58.989140 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:58.989107 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:59.007131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.007108 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.007249 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.007139 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5sd\" (UniqueName: \"kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.007249 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.007167 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.007249 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.007205 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.007443 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:59.007281 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 20:48:59.007443 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:48:59.007329 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs podName:b8d7c0ca-4729-418f-a1ef-04f34dbc651e nodeName:}" failed. No retries permitted until 2026-10-30 20:48:59.507315401 +0000 UTC m=+767.614811290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "b8d7c0ca-4729-418f-a1ef-04f34dbc651e") : secret "data-science-collector-collector-headless-tls" not found Oct 30 20:48:59.007883 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.007860 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.009600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.009577 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.069254 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.069220 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5sd\" (UniqueName: \"kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.138305 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.138267 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:59.512747 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.512656 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.515064 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.515031 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:48:59.772168 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:48:59.772080 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:00.145900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:00.145869 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:00.149438 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:49:00.149390 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d7c0ca_4729_418f_a1ef_04f34dbc651e.slice/crio-e7086bd5901c2d33ad40f0ca9a4e6c8fc5c29abb14fe3d161b5bb1720c668238 WatchSource:0}: Error finding container e7086bd5901c2d33ad40f0ca9a4e6c8fc5c29abb14fe3d161b5bb1720c668238: Status 404 returned error can't find the container with id e7086bd5901c2d33ad40f0ca9a4e6c8fc5c29abb14fe3d161b5bb1720c668238 Oct 30 20:49:00.151380 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:00.151346 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:49:00.151484 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:00.151427 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:49:01.105801 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:01.105760 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b8d7c0ca-4729-418f-a1ef-04f34dbc651e","Type":"ContainerStarted","Data":"b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0"} Oct 30 20:49:01.106007 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:01.105811 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b8d7c0ca-4729-418f-a1ef-04f34dbc651e","Type":"ContainerStarted","Data":"e7086bd5901c2d33ad40f0ca9a4e6c8fc5c29abb14fe3d161b5bb1720c668238"} Oct 30 20:49:01.224944 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:01.224889 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.224874163 podStartE2EDuration="3.224874163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:01.223878094 +0000 UTC m=+769.331374006" watchObservedRunningTime="2026-10-30 20:49:01.224874163 +0000 UTC m=+769.332370137" Oct 30 20:49:05.725516 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:05.725477 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:21.308161 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.308123 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:21.308621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.308312 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b8d7c0ca-4729-418f-a1ef-04f34dbc651e" containerName="otc-container" containerID="cri-o://b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0" gracePeriod=30 Oct 30 20:49:21.362025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.361994 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:21.510553 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.510522 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:21.552799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.552764 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:21.704738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.704709 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd\" (UniqueName: \"kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd\") pod \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " Oct 30 20:49:21.704917 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.704762 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\") pod \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " Oct 30 20:49:21.704917 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.704818 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs\") pod \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " Oct 30 20:49:21.704917 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.704852 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal\") pod \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\" (UID: \"b8d7c0ca-4729-418f-a1ef-04f34dbc651e\") " Oct 30 20:49:21.705236 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.705212 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal" pod "b8d7c0ca-4729-418f-a1ef-04f34dbc651e" (UID: "b8d7c0ca-4729-418f-a1ef-04f34dbc651e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:49:21.706894 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.706860 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs" pod "b8d7c0ca-4729-418f-a1ef-04f34dbc651e" (UID: "b8d7c0ca-4729-418f-a1ef-04f34dbc651e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:49:21.706894 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.706873 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd" pod "b8d7c0ca-4729-418f-a1ef-04f34dbc651e" (UID: "b8d7c0ca-4729-418f-a1ef-04f34dbc651e"). InnerVolumeSpecName "kube-api-access-2s5sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:49:21.707043 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.706947 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs" pod "b8d7c0ca-4729-418f-a1ef-04f34dbc651e" (UID: "b8d7c0ca-4729-418f-a1ef-04f34dbc651e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:49:21.806304 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.806265 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:21.806304 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.806297 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s5sd\" (UniqueName: \"kubernetes.io/projected/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-kube-api-access-2s5sd\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:21.806304 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.806307 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:21.806546 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:21.806316 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d7c0ca-4729-418f-a1ef-04f34dbc651e-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:22.045432 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.045372 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:22.194882 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.194845 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:22.197588 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.197564 4988 generic.go:356] "Generic (PLEG): container finished" podID="b8d7c0ca-4729-418f-a1ef-04f34dbc651e" containerID="b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0" exitCode=0 Oct 30 20:49:22.197730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.197616 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:22.197730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.197635 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b8d7c0ca-4729-418f-a1ef-04f34dbc651e","Type":"ContainerDied","Data":"b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0"} Oct 30 20:49:22.197730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.197674 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b8d7c0ca-4729-418f-a1ef-04f34dbc651e","Type":"ContainerDied","Data":"e7086bd5901c2d33ad40f0ca9a4e6c8fc5c29abb14fe3d161b5bb1720c668238"} Oct 30 20:49:22.197730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.197694 4988 scope.go:122] "RemoveContainer" containerID="b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0" Oct 30 20:49:22.205700 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.205681 4988 scope.go:122] "RemoveContainer" containerID="b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0" Oct 30 20:49:22.205966 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:49:22.205940 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0\": container with ID starting with b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0 not found: ID does not exist" containerID="b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0" Oct 30 20:49:22.206018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.205974 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0"} err="failed to get container status \"b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0\": rpc error: code = NotFound desc = could not find container \"b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0\": container with ID starting with b8488ef731a6030eb89c513731f09c66e2a466848520c9794c7920bb9e2111b0 not found: ID does not exist" Oct 30 20:49:22.361997 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.361921 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:22.411503 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:22.411474 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:24.014920 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:24.014890 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d7c0ca-4729-418f-a1ef-04f34dbc651e" path="/var/lib/kubelet/pods/b8d7c0ca-4729-418f-a1ef-04f34dbc651e/volumes" Oct 30 20:49:26.393042 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:49:26.393008 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:49:28.192195 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:28.192159 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:28.390568 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:28.390522 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:28.540680 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:28.540599 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:28.685378 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:28.685343 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:31.783099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.783054 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:31.818221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.818192 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:31.818599 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.818582 4988 cpu_manager.go:414] "containerMap: removing container" podUID="b8d7c0ca-4729-418f-a1ef-04f34dbc651e" containerName="otc-container" Oct 30 20:49:31.818692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.818600 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d7c0ca-4729-418f-a1ef-04f34dbc651e" containerName="otc-container" Oct 30 20:49:31.818746 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.818689 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d7c0ca-4729-418f-a1ef-04f34dbc651e" containerName="otc-container" Oct 30 20:49:31.821726 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.821707 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.872088 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.872059 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkh4\" (UniqueName: \"kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.872234 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.872093 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.872234 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.872113 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.872306 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.872261 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.872306 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.872291 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.881983 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.881956 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:31.923375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.923351 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:49:31.930048 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.930021 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:49:31.930167 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.930143 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-4d267ad2\"" Oct 30 20:49:31.930222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.930162 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9htg9\"" Oct 30 20:49:31.930222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.930206 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:49:31.930222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.930215 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:49:31.930322 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.930222 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:49:31.945508 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.945472 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:31.973105 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.973080 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.973200 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.973120 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.973200 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.973152 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkh4\" (UniqueName: \"kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.973200 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.973172 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.973360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.973204 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.973915 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.973892 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.975492 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.975474 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.975882 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.975866 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:31.975968 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:31.975947 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:32.034724 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.034663 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkh4\" (UniqueName: \"kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4\") pod \"data-science-collector-collector-1\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:32.097044 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.097009 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:32.133885 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.133862 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:32.244825 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.244793 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:32.504951 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.504919 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:32.536802 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:49:32.536778 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab64f4f3_3922_4ce8_a300_78bdfa73ba9b.slice/crio-537bd2ccc62dabf3c32dd9fa125a00b3da61478e9c0835dfa522df8d34750361 WatchSource:0}: Error finding container 537bd2ccc62dabf3c32dd9fa125a00b3da61478e9c0835dfa522df8d34750361: Status 404 returned error can't find the container with id 537bd2ccc62dabf3c32dd9fa125a00b3da61478e9c0835dfa522df8d34750361 Oct 30 20:49:32.538713 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.538682 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:49:32.538814 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:32.538737 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:49:33.245714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:33.245680 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b","Type":"ContainerStarted","Data":"4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba"} Oct 30 20:49:33.246087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:33.245733 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b","Type":"ContainerStarted","Data":"537bd2ccc62dabf3c32dd9fa125a00b3da61478e9c0835dfa522df8d34750361"} Oct 30 20:49:33.364250 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:33.364198 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.364184802 podStartE2EDuration="2.364184802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:33.362959468 +0000 UTC m=+801.470455390" watchObservedRunningTime="2026-10-30 20:49:33.364184802 +0000 UTC m=+801.471680796" Oct 30 20:49:50.193393 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:50.193356 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:50.391184 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:50.391151 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.392153 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:49:56.392115 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:49:58.695475 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.695436 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:58.695867 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.695657 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" containerName="otc-container" containerID="cri-o://4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba" gracePeriod=30 Oct 30 20:49:58.729456 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.729402 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.937026 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.937003 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:58.938395 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.938370 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.975082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.975022 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs\") pod \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " Oct 30 20:49:58.975230 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.975096 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert\") pod \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " Oct 30 20:49:58.975230 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.975140 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4\" (UniqueName: \"kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4\") pod \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " Oct 30 20:49:58.975230 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.975194 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs\") pod \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " Oct 30 20:49:58.975390 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.975240 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal\") pod \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\" (UID: \"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b\") " Oct 30 20:49:58.975739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.975709 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal" pod "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" (UID: "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:49:58.977151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.977126 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert" pod "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" (UID: "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:49:58.977251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.977210 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs" pod "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" (UID: "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:49:58.977574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.977555 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs" pod "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" (UID: "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:49:58.977641 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:58.977571 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4" pod "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" (UID: "ab64f4f3-3922-4ce8-a300-78bdfa73ba9b"). InnerVolumeSpecName "kube-api-access-fmkh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:49:59.076286 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.076249 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:59.076702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.076288 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkh4\" (UniqueName: \"kubernetes.io/projected/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-kube-api-access-fmkh4\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:59.076702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.076307 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:59.076702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.076321 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:59.076702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.076335 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:49:59.349518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.349437 4988 generic.go:356] "Generic (PLEG): container finished" podID="ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" containerID="4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba" exitCode=0 Oct 30 20:49:59.349518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.349483 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:49:59.349518 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.349491 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b","Type":"ContainerDied","Data":"4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba"} Oct 30 20:49:59.349747 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.349530 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab64f4f3-3922-4ce8-a300-78bdfa73ba9b","Type":"ContainerDied","Data":"537bd2ccc62dabf3c32dd9fa125a00b3da61478e9c0835dfa522df8d34750361"} Oct 30 20:49:59.349747 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.349551 4988 scope.go:122] "RemoveContainer" containerID="4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba" Oct 30 20:49:59.358305 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.358286 4988 scope.go:122] "RemoveContainer" containerID="4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba" Oct 30 20:49:59.358566 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:49:59.358550 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba\": container with ID starting with 4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba not found: ID does not exist" containerID="4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba" Oct 30 20:49:59.358617 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.358575 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba"} err="failed to get container status \"4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba\": rpc error: code = NotFound desc = could not find container \"4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba\": container with ID starting with 4e57c0e8e7c0a3cb8cc9754945c6e1edf84bd2c4143b346f3768a10bc00a6cba not found: ID does not exist" Oct 30 20:49:59.516487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.516457 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:59.523823 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.523794 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:59.561129 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.561100 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:49:59.721735 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:49:59.721696 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:00.015976 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:00.015905 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" path="/var/lib/kubelet/pods/ab64f4f3-3922-4ce8-a300-78bdfa73ba9b/volumes" Oct 30 20:50:06.081953 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:06.081916 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.588929 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:12.588890 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.787071 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:12.787042 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.940598 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:12.940563 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.114532 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.114485 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.281276 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.281190 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.458004 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.457968 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.693884 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.693845 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.780889 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.780856 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv"] Oct 30 20:50:13.781196 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.781183 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" containerName="otc-container" Oct 30 20:50:13.781237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.781197 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" containerName="otc-container" Oct 30 20:50:13.781291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.781282 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab64f4f3-3922-4ce8-a300-78bdfa73ba9b" containerName="otc-container" Oct 30 20:50:13.785862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.785844 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.846929 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.846905 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv"] Oct 30 20:50:13.893636 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.893608 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.893777 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.893667 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.893777 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.893748 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdsm\" (UniqueName: \"kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.894434 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.894392 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-q67wt\"" Oct 30 20:50:13.894434 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.894401 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 20:50:13.894541 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.894395 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 20:50:13.894541 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.894395 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:50:13.894541 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.894430 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:50:13.898189 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.898161 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.995039 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.994967 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.995039 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.995005 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdsm\" (UniqueName: \"kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.995039 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.995040 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.995692 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.995666 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:13.997393 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:13.997363 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:14.056322 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:14.056289 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdsm\" (UniqueName: \"kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm\") pod \"data-science-thanos-querier-proxy-77657866bd-672qv\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:14.101822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:14.101795 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:50:14.475184 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:14.475149 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv"] Oct 30 20:50:14.479147 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:50:14.479118 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1ffa1d_6902_4428_9cfe_c4948086419d.slice/crio-b4c91b84f31f65be28b63add6f9db0cda2cd04bbcd2d0deee6a1da22385c1bfa WatchSource:0}: Error finding container b4c91b84f31f65be28b63add6f9db0cda2cd04bbcd2d0deee6a1da22385c1bfa: Status 404 returned error can't find the container with id b4c91b84f31f65be28b63add6f9db0cda2cd04bbcd2d0deee6a1da22385c1bfa Oct 30 20:50:14.481050 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:14.481027 4988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:50:15.318941 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:15.318907 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:15.413007 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:15.412973 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerStarted","Data":"b4c91b84f31f65be28b63add6f9db0cda2cd04bbcd2d0deee6a1da22385c1bfa"} Oct 30 20:50:18.468962 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.468930 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:50:18.472237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.472219 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.480749 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.480722 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:18.535626 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.535597 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:50:18.557592 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.557562 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:50:18.583822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.583366 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:50:18.583822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.583528 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 20:50:18.583822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.583620 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:50:18.583822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.583677 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7dpdl\"" Oct 30 20:50:18.583822 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.583791 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:50:18.632153 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.632116 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.632316 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.632163 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwb85\" (UniqueName: \"kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.632316 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.632196 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.632316 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.632252 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.632481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.632350 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.632481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.632394 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.689854 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.689816 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:18.733741 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.733653 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.733741 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.733719 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.733968 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.733805 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.733968 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.733903 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwb85\" (UniqueName: \"kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.733968 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.733940 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.734123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.733965 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.734725 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.734697 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.736450 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.736404 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.736628 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.736607 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.736996 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.736975 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.737071 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.737006 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.794632 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.794597 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwb85\" (UniqueName: \"kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85\") pod \"data-science-collector-collector-1\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:18.848754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.848719 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:18.997618 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:18.997541 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:19.085857 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:19.085832 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:50:19.459355 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:19.459323 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:50:19.462329 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:50:19.462304 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b931a9_06ed_4af3_86c7_0cee62477555.slice/crio-626d5f3a6f6fe9e7614b729144c033209ec0418666a4c219fbfea6fd5db7d3f7 WatchSource:0}: Error finding container 626d5f3a6f6fe9e7614b729144c033209ec0418666a4c219fbfea6fd5db7d3f7: Status 404 returned error can't find the container with id 626d5f3a6f6fe9e7614b729144c033209ec0418666a4c219fbfea6fd5db7d3f7 Oct 30 20:50:19.464527 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:19.464499 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:50:19.464633 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:19.464550 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:50:20.437304 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:20.437264 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c4b931a9-06ed-4af3-86c7-0cee62477555","Type":"ContainerStarted","Data":"4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99"} Oct 30 20:50:20.437716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:20.437326 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c4b931a9-06ed-4af3-86c7-0cee62477555","Type":"ContainerStarted","Data":"626d5f3a6f6fe9e7614b729144c033209ec0418666a4c219fbfea6fd5db7d3f7"} Oct 30 20:50:20.591021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:20.590952 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.5909325069999998 podStartE2EDuration="2.590932507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:20.556701658 +0000 UTC m=+848.664197568" watchObservedRunningTime="2026-10-30 20:50:20.590932507 +0000 UTC m=+848.698428417" Oct 30 20:50:21.442980 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:21.442946 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerStarted","Data":"2d3a3e86ad73046a203e60ea18872c805de265e1536f8efff7405a3cfbe8f286"} Oct 30 20:50:24.460467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:24.460433 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerStarted","Data":"ee39c83aa83d2ca33c64845f981663cdee0288d928f80d3d1f24607091e06956"} Oct 30 20:50:25.466489 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:25.466450 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerStarted","Data":"9644de243969e1faeff8480e8fd06cb0e449ec81655702386d01829cb45e25f4"} Oct 30 20:50:25.587222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:25.587100 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" podStartSLOduration=1.833083257 podStartE2EDuration="12.587072463s" totalImagesPullingTime="10.753989206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:25.584889107 +0000 UTC m=+853.692385017" watchObservedRunningTime="2026-10-30 20:50:25.587072463 +0000 UTC m=+853.694568374" Oct 30 20:50:26.391713 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:50:26.391672 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:50:30.800865 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:30.800834 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx"] Oct 30 20:50:30.804320 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:30.804299 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:50:30.856620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:30.856601 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-48j4p\"" Oct 30 20:50:30.907102 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:30.907076 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx"] Oct 30 20:50:30.935685 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:30.935656 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrz5\" (UniqueName: \"kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx\" (UID: \"c8b333c4-fbd6-42ae-8718-80664f887846\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:50:31.036697 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.036671 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrz5\" (UniqueName: \"kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx\" (UID: \"c8b333c4-fbd6-42ae-8718-80664f887846\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:50:31.067728 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.067667 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:31.098190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.098163 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrz5\" (UniqueName: \"kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx\" (UID: \"c8b333c4-fbd6-42ae-8718-80664f887846\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:50:31.115060 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.115039 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:50:31.488429 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.488373 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx"] Oct 30 20:50:31.491915 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:50:31.491885 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8b333c4_fbd6_42ae_8718_80664f887846.slice/crio-10a63e1d34340d21b2e61451468865eff3d323e5807201da87dd9c0597d75511 WatchSource:0}: Error finding container 10a63e1d34340d21b2e61451468865eff3d323e5807201da87dd9c0597d75511: Status 404 returned error can't find the container with id 10a63e1d34340d21b2e61451468865eff3d323e5807201da87dd9c0597d75511 Oct 30 20:50:31.496937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.496914 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:31.647658 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.647626 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:31.800650 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:31.800567 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.369782 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.369746 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.496357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.496326 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" event={"ID":"c8b333c4-fbd6-42ae-8718-80664f887846","Type":"ContainerStarted","Data":"10a63e1d34340d21b2e61451468865eff3d323e5807201da87dd9c0597d75511"} Oct 30 20:50:32.531270 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.531238 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:50:32.535784 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.535767 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.576996 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.576968 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.619913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.619858 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 20:50:32.638670 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.638643 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 20:50:32.638823 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.638804 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 20:50:32.638871 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.638814 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 20:50:32.638905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.638877 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-l4l28\"" Oct 30 20:50:32.640131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.640113 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:50:32.651261 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651238 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.651363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651278 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.651363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651292 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.651469 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651391 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.651520 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651487 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.651556 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651525 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj426\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.651591 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.651553 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.735813 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.735781 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.752263 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752237 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.752354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752268 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj426\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.752354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752291 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.752499 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752477 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.752602 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752580 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.752738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752616 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.752738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.752671 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.753122 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.753086 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.754847 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.754823 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.754945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.754859 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.754945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.754859 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.754945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.754935 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.755151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.755064 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.814115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.814093 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj426\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.853067 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.853043 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:32.935267 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:32.935232 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:33.320153 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:33.320005 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:50:33.323600 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:50:33.323569 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee07f7c6_92e0_4599_842a_eec061e7439e.slice/crio-9bb397d51ab2551fa900f5a93f9e2ce81a1c5259f6e36552865d18afd6e7cd52 WatchSource:0}: Error finding container 9bb397d51ab2551fa900f5a93f9e2ce81a1c5259f6e36552865d18afd6e7cd52: Status 404 returned error can't find the container with id 9bb397d51ab2551fa900f5a93f9e2ce81a1c5259f6e36552865d18afd6e7cd52 Oct 30 20:50:33.503746 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:33.503713 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerStarted","Data":"9bb397d51ab2551fa900f5a93f9e2ce81a1c5259f6e36552865d18afd6e7cd52"} Oct 30 20:50:35.057749 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.057714 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:35.513782 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.513747 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" event={"ID":"c8b333c4-fbd6-42ae-8718-80664f887846","Type":"ContainerStarted","Data":"6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6"} Oct 30 20:50:35.686542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.686487 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" podStartSLOduration=1.8585388699999998 podStartE2EDuration="5.686467952s" totalImagesPullingTime="3.827929082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:35.636264129 +0000 UTC m=+863.743760043" watchObservedRunningTime="2026-10-30 20:50:35.686467952 +0000 UTC m=+863.793963917" Oct 30 20:50:35.687816 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.687791 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:50:35.692290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.692273 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.750103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.750078 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 20:50:35.750300 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.750282 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 20:50:35.750391 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.750300 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 20:50:35.754168 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.754146 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 20:50:35.801024 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.800999 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:50:35.854205 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.854181 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 20:50:35.854787 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.854761 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 20:50:35.854787 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.854768 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 20:50:35.854931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.854830 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 20:50:35.855013 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.854999 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 20:50:35.855075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.855057 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-87trm\"" Oct 30 20:50:35.981314 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981241 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981314 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981291 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981317 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw67f\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981436 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981466 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981498 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981687 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981611 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981719 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981688 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981756 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981731 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981797 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981767 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981836 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981800 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:35.981869 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:35.981829 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.082938 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.082907 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.082964 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.082997 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083025 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083054 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083096 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083130 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083155 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw67f\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083183 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083208 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083239 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.083375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.083283 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.084394 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.084361 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.084798 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.084766 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.085331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.085303 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.086194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.086152 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.086623 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.086600 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.087509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.087464 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.087606 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.087567 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.087907 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.087886 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.088082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.088061 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.088344 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.088319 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.089123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.089099 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.145428 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.145382 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw67f\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:36.186393 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:36.186364 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:37.523825 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:37.523744 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerStarted","Data":"ec3ca33346b7bc675ddc2ca09222c528bfc097510b5325a8ada0baff45c562f2"} Oct 30 20:50:40.537399 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:40.537365 4988 generic.go:356] "Generic (PLEG): container finished" podID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerID="ec3ca33346b7bc675ddc2ca09222c528bfc097510b5325a8ada0baff45c562f2" exitCode=0 Oct 30 20:50:40.537817 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:40.537437 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerDied","Data":"ec3ca33346b7bc675ddc2ca09222c528bfc097510b5325a8ada0baff45c562f2"} Oct 30 20:50:43.036150 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.036112 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.141373 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.141337 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.237848 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.237811 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.238023 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.237955 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.291464 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.291376 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") DevicePath \"csi-468025a65b88095496ffee260c47731d5a0a7d1245692f39a473b974fd0f0891\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.496231 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.496201 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fb02f91ab75afdcfd8c2ac3eaeb0ba7303c67490d38fe3164f5fc7856f2f1875/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.562757 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.562680 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:43.812005 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:43.811971 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:44.338444 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:50:44.338385 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7d2ea7_6dfe_41b7_8692_d6e9ec21727d.slice/crio-2d9c02a00f2df934b526d68e9fb8156ba90155c5634c4c13f33f4e58101e3325 WatchSource:0}: Error finding container 2d9c02a00f2df934b526d68e9fb8156ba90155c5634c4c13f33f4e58101e3325: Status 404 returned error can't find the container with id 2d9c02a00f2df934b526d68e9fb8156ba90155c5634c4c13f33f4e58101e3325 Oct 30 20:50:44.358081 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:44.358056 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:50:44.559190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:44.559102 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerStarted","Data":"b1fb6d388761f7432f1ff0b8f3504b805cbbd3a3835cfc9d9d2a416fdcad9595"} Oct 30 20:50:44.560191 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:44.560165 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerStarted","Data":"2d9c02a00f2df934b526d68e9fb8156ba90155c5634c4c13f33f4e58101e3325"} Oct 30 20:50:45.565823 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:45.565789 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerStarted","Data":"13d87c59dc329c04e4fc286cf36708252552e6fe87b91343d3bc02ce3fbb58b9"} Oct 30 20:50:45.566312 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:45.565891 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:45.567302 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:45.567281 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerStarted","Data":"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47"} Oct 30 20:50:45.788928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:45.788867 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.95677132 podStartE2EDuration="13.788852966s" totalImagesPullingTime="6.545681523s" totalInitContainerRuntime="3.286400123s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:45.787567066 +0000 UTC m=+873.895062980" watchObservedRunningTime="2026-10-30 20:50:45.788852966 +0000 UTC m=+873.896348908" Oct 30 20:50:48.583564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:48.583528 4988 generic.go:356] "Generic (PLEG): container finished" podID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerID="f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47" exitCode=0 Oct 30 20:50:48.583969 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:48.583572 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerDied","Data":"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47"} Oct 30 20:50:51.577172 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:51.577140 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:50:55.617271 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:55.617240 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerStarted","Data":"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9"} Oct 30 20:50:56.393011 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:50:56.392978 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:50:56.623784 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:56.623747 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerStarted","Data":"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821"} Oct 30 20:50:56.624185 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:56.623823 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerStarted","Data":"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab"} Oct 30 20:50:56.884217 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:56.884159 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=14.979311749 podStartE2EDuration="24.884144335s" totalImagesPullingTime="6.819384765s" totalInitContainerRuntime="3.085447821s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:56.834660398 +0000 UTC m=+884.942156308" watchObservedRunningTime="2026-10-30 20:50:56.884144335 +0000 UTC m=+884.991640250" Oct 30 20:50:58.812159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:58.812126 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:58.812590 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:58.812193 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:58.817564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:58.817542 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:50:59.648128 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:50:59.648104 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:51:06.424146 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:06.424108 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:11.507971 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.507935 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv"] Oct 30 20:51:11.508445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.508251 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="kube-rbac-proxy" containerID="cri-o://2d3a3e86ad73046a203e60ea18872c805de265e1536f8efff7405a3cfbe8f286" gracePeriod=30 Oct 30 20:51:11.508445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.508287 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="prom-label-proxy" containerID="cri-o://9644de243969e1faeff8480e8fd06cb0e449ec81655702386d01829cb45e25f4" gracePeriod=30 Oct 30 20:51:11.508445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.508317 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="method-gate" containerID="cri-o://ee39c83aa83d2ca33c64845f981663cdee0288d928f80d3d1f24607091e06956" gracePeriod=30 Oct 30 20:51:11.696544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.696501 4988 generic.go:356] "Generic (PLEG): container finished" podID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerID="9644de243969e1faeff8480e8fd06cb0e449ec81655702386d01829cb45e25f4" exitCode=0 Oct 30 20:51:11.696544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.696535 4988 generic.go:356] "Generic (PLEG): container finished" podID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerID="ee39c83aa83d2ca33c64845f981663cdee0288d928f80d3d1f24607091e06956" exitCode=0 Oct 30 20:51:11.696544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.696545 4988 generic.go:356] "Generic (PLEG): container finished" podID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerID="2d3a3e86ad73046a203e60ea18872c805de265e1536f8efff7405a3cfbe8f286" exitCode=0 Oct 30 20:51:11.696804 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.696550 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerDied","Data":"9644de243969e1faeff8480e8fd06cb0e449ec81655702386d01829cb45e25f4"} Oct 30 20:51:11.696804 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.696647 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerDied","Data":"ee39c83aa83d2ca33c64845f981663cdee0288d928f80d3d1f24607091e06956"} Oct 30 20:51:11.696804 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.696691 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerDied","Data":"2d3a3e86ad73046a203e60ea18872c805de265e1536f8efff7405a3cfbe8f286"} Oct 30 20:51:11.748183 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.748155 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:51:11.761402 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.761343 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm\" (UniqueName: \"kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm\") pod \"ff1ffa1d-6902-4428-9cfe-c4948086419d\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " Oct 30 20:51:11.761538 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.761398 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls\" (UniqueName: \"kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls\") pod \"ff1ffa1d-6902-4428-9cfe-c4948086419d\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " Oct 30 20:51:11.761538 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.761441 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config\" (UniqueName: \"kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config\") pod \"ff1ffa1d-6902-4428-9cfe-c4948086419d\" (UID: \"ff1ffa1d-6902-4428-9cfe-c4948086419d\") " Oct 30 20:51:11.761785 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.761761 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config" pod "ff1ffa1d-6902-4428-9cfe-c4948086419d" (UID: "ff1ffa1d-6902-4428-9cfe-c4948086419d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:11.763236 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.763218 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm" pod "ff1ffa1d-6902-4428-9cfe-c4948086419d" (UID: "ff1ffa1d-6902-4428-9cfe-c4948086419d"). InnerVolumeSpecName "kube-api-access-4rdsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:11.763334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.763317 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls" pod "ff1ffa1d-6902-4428-9cfe-c4948086419d" (UID: "ff1ffa1d-6902-4428-9cfe-c4948086419d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:11.862775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.862741 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ff1ffa1d-6902-4428-9cfe-c4948086419d-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:11.862775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.862768 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff1ffa1d-6902-4428-9cfe-c4948086419d-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:11.862775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:11.862778 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rdsm\" (UniqueName: \"kubernetes.io/projected/ff1ffa1d-6902-4428-9cfe-c4948086419d-kube-api-access-4rdsm\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:12.093072 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.093043 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:51:12.096523 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.096500 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:51:12.455461 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.455426 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx"] Oct 30 20:51:12.455966 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.455921 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" podUID="c8b333c4-fbd6-42ae-8718-80664f887846" containerName="thanos-querier" containerID="cri-o://6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6" gracePeriod=30 Oct 30 20:51:12.506812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.506780 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.699980 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.699954 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:51:12.702669 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.702645 4988 generic.go:356] "Generic (PLEG): container finished" podID="c8b333c4-fbd6-42ae-8718-80664f887846" containerID="6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6" exitCode=0 Oct 30 20:51:12.702794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.702702 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" Oct 30 20:51:12.702794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.702703 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.702794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.702749 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" event={"ID":"c8b333c4-fbd6-42ae-8718-80664f887846","Type":"ContainerDied","Data":"6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6"} Oct 30 20:51:12.702794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.702785 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx" event={"ID":"c8b333c4-fbd6-42ae-8718-80664f887846","Type":"ContainerDied","Data":"10a63e1d34340d21b2e61451468865eff3d323e5807201da87dd9c0597d75511"} Oct 30 20:51:12.702982 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.702807 4988 scope.go:122] "RemoveContainer" containerID="6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6" Oct 30 20:51:12.704676 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.704655 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" event={"ID":"ff1ffa1d-6902-4428-9cfe-c4948086419d","Type":"ContainerDied","Data":"b4c91b84f31f65be28b63add6f9db0cda2cd04bbcd2d0deee6a1da22385c1bfa"} Oct 30 20:51:12.704786 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.704676 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv" Oct 30 20:51:12.710811 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.710786 4988 scope.go:122] "RemoveContainer" containerID="6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6" Oct 30 20:51:12.711058 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:12.711041 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6\": container with ID starting with 6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6 not found: ID does not exist" containerID="6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6" Oct 30 20:51:12.711106 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.711066 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6"} err="failed to get container status \"6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6\": rpc error: code = NotFound desc = could not find container \"6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6\": container with ID starting with 6b781d523a7b81c2da6b1cb22330d10e6c862e7693f562d95492c02073e01dc6 not found: ID does not exist" Oct 30 20:51:12.711106 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.711082 4988 scope.go:122] "RemoveContainer" containerID="9644de243969e1faeff8480e8fd06cb0e449ec81655702386d01829cb45e25f4" Oct 30 20:51:12.720784 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.720763 4988 scope.go:122] "RemoveContainer" containerID="ee39c83aa83d2ca33c64845f981663cdee0288d928f80d3d1f24607091e06956" Oct 30 20:51:12.728875 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.728855 4988 scope.go:122] "RemoveContainer" containerID="2d3a3e86ad73046a203e60ea18872c805de265e1536f8efff7405a3cfbe8f286" Oct 30 20:51:12.770289 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.770259 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5\" (UniqueName: \"kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5\") pod \"c8b333c4-fbd6-42ae-8718-80664f887846\" (UID: \"c8b333c4-fbd6-42ae-8718-80664f887846\") " Oct 30 20:51:12.772395 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.772363 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5" pod "c8b333c4-fbd6-42ae-8718-80664f887846" (UID: "c8b333c4-fbd6-42ae-8718-80664f887846"). InnerVolumeSpecName "kube-api-access-7vrz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:12.798730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.798706 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:51:12.798950 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.798930 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="alertmanager" containerID="cri-o://b1fb6d388761f7432f1ff0b8f3504b805cbbd3a3835cfc9d9d2a416fdcad9595" gracePeriod=120 Oct 30 20:51:12.799035 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.799007 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="config-reloader" containerID="cri-o://13d87c59dc329c04e4fc286cf36708252552e6fe87b91343d3bc02ce3fbb58b9" gracePeriod=120 Oct 30 20:51:12.816029 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.816005 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:51:12.816290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.816263 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="prometheus" containerID="cri-o://c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" gracePeriod=600 Oct 30 20:51:12.816383 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.816275 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="config-reloader" containerID="cri-o://804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" gracePeriod=600 Oct 30 20:51:12.816383 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.816263 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="thanos-sidecar" containerID="cri-o://a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" gracePeriod=600 Oct 30 20:51:12.863084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.863055 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.871843 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.871817 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vrz5\" (UniqueName: \"kubernetes.io/projected/c8b333c4-fbd6-42ae-8718-80664f887846-kube-api-access-7vrz5\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:12.950914 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.950888 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv"] Oct 30 20:51:12.958368 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:12.958347 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-672qv"] Oct 30 20:51:13.028372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.028345 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:13.054255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.054237 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:51:13.073669 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073644 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.073775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073681 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.073775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073704 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.073775 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073723 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.073941 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073822 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.073941 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073857 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.073941 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073903 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.074097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.073947 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.074097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074048 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.074097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074083 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.074243 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074125 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:13.074295 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074139 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.074344 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074307 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:13.074344 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074327 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.074735 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074710 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:13.074846 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.074826 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out\") pod \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\" (UID: \"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d\") " Oct 30 20:51:13.075166 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.075144 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.075239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.075173 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.075239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.075189 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.076743 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.076719 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:13.076879 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.076843 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.077112 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.077084 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.077184 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.077143 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.077425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.077388 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "kube-api-access-fw67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:13.077578 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.077560 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.078395 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.078370 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.078724 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.078704 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:13.079787 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.079766 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "pvc-e73afc79-aa37-49a0-8744-633c03cad1f4". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:51:13.088999 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.088978 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config" pod "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" (UID: "8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.175574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175546 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175573 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175586 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175600 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175616 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175629 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175643 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175657 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw67f\" (UniqueName: \"kubernetes.io/projected/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-kube-api-access-fw67f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175672 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.175708 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.175704 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:51:13.176135 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.176023 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:13.186618 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.186603 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e73afc79-aa37-49a0-8744-633c03cad1f4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:51:13.277018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.276970 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-468025a65b88095496ffee260c47731d5a0a7d1245692f39a473b974fd0f0891\"" Oct 30 20:51:13.508126 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.508098 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx"] Oct 30 20:51:13.512859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.512839 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-w45fx"] Oct 30 20:51:13.712016 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.711989 4988 generic.go:356] "Generic (PLEG): container finished" podID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerID="13d87c59dc329c04e4fc286cf36708252552e6fe87b91343d3bc02ce3fbb58b9" exitCode=0 Oct 30 20:51:13.712016 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.712012 4988 generic.go:356] "Generic (PLEG): container finished" podID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerID="b1fb6d388761f7432f1ff0b8f3504b805cbbd3a3835cfc9d9d2a416fdcad9595" exitCode=0 Oct 30 20:51:13.712471 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.712022 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerDied","Data":"13d87c59dc329c04e4fc286cf36708252552e6fe87b91343d3bc02ce3fbb58b9"} Oct 30 20:51:13.712471 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.712082 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerDied","Data":"b1fb6d388761f7432f1ff0b8f3504b805cbbd3a3835cfc9d9d2a416fdcad9595"} Oct 30 20:51:13.714623 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714598 4988 generic.go:356] "Generic (PLEG): container finished" podID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerID="a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" exitCode=0 Oct 30 20:51:13.714712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714623 4988 generic.go:356] "Generic (PLEG): container finished" podID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerID="804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" exitCode=0 Oct 30 20:51:13.714712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714632 4988 generic.go:356] "Generic (PLEG): container finished" podID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerID="c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" exitCode=0 Oct 30 20:51:13.714712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714664 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:51:13.714712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714685 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerDied","Data":"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821"} Oct 30 20:51:13.714912 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714760 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerDied","Data":"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab"} Oct 30 20:51:13.714912 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714803 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerDied","Data":"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9"} Oct 30 20:51:13.714912 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714840 4988 scope.go:122] "RemoveContainer" containerID="a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" Oct 30 20:51:13.714912 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.714842 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d","Type":"ContainerDied","Data":"2d9c02a00f2df934b526d68e9fb8156ba90155c5634c4c13f33f4e58101e3325"} Oct 30 20:51:13.733813 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.733794 4988 scope.go:122] "RemoveContainer" containerID="804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" Oct 30 20:51:13.736397 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.736377 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:51:13.742378 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.742356 4988 scope.go:122] "RemoveContainer" containerID="c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" Oct 30 20:51:13.749633 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.749617 4988 scope.go:122] "RemoveContainer" containerID="f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47" Oct 30 20:51:13.756816 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.756794 4988 scope.go:122] "RemoveContainer" containerID="a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" Oct 30 20:51:13.757064 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:13.757045 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": container with ID starting with a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821 not found: ID does not exist" containerID="a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" Oct 30 20:51:13.757135 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757073 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821"} err="failed to get container status \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": rpc error: code = NotFound desc = could not find container \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": container with ID starting with a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821 not found: ID does not exist" Oct 30 20:51:13.757135 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757092 4988 scope.go:122] "RemoveContainer" containerID="804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" Oct 30 20:51:13.757332 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:13.757315 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": container with ID starting with 804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab not found: ID does not exist" containerID="804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" Oct 30 20:51:13.757389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757344 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab"} err="failed to get container status \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": rpc error: code = NotFound desc = could not find container \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": container with ID starting with 804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab not found: ID does not exist" Oct 30 20:51:13.757389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757366 4988 scope.go:122] "RemoveContainer" containerID="c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" Oct 30 20:51:13.757637 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:13.757620 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": container with ID starting with c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9 not found: ID does not exist" containerID="c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" Oct 30 20:51:13.757673 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757641 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9"} err="failed to get container status \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": rpc error: code = NotFound desc = could not find container \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": container with ID starting with c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9 not found: ID does not exist" Oct 30 20:51:13.757673 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757656 4988 scope.go:122] "RemoveContainer" containerID="f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47" Oct 30 20:51:13.757904 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:13.757882 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": container with ID starting with f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47 not found: ID does not exist" containerID="f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47" Oct 30 20:51:13.757979 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757910 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47"} err="failed to get container status \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": rpc error: code = NotFound desc = could not find container \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": container with ID starting with f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47 not found: ID does not exist" Oct 30 20:51:13.757979 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.757927 4988 scope.go:122] "RemoveContainer" containerID="a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" Oct 30 20:51:13.758177 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758161 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821"} err="failed to get container status \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": rpc error: code = NotFound desc = could not find container \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": container with ID starting with a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821 not found: ID does not exist" Oct 30 20:51:13.758235 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758178 4988 scope.go:122] "RemoveContainer" containerID="804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" Oct 30 20:51:13.758404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758386 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab"} err="failed to get container status \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": rpc error: code = NotFound desc = could not find container \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": container with ID starting with 804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab not found: ID does not exist" Oct 30 20:51:13.758404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758404 4988 scope.go:122] "RemoveContainer" containerID="c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" Oct 30 20:51:13.758649 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758627 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9"} err="failed to get container status \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": rpc error: code = NotFound desc = could not find container \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": container with ID starting with c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9 not found: ID does not exist" Oct 30 20:51:13.758739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758649 4988 scope.go:122] "RemoveContainer" containerID="f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47" Oct 30 20:51:13.758851 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758833 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47"} err="failed to get container status \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": rpc error: code = NotFound desc = could not find container \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": container with ID starting with f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47 not found: ID does not exist" Oct 30 20:51:13.758900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.758851 4988 scope.go:122] "RemoveContainer" containerID="a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821" Oct 30 20:51:13.759053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759035 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821"} err="failed to get container status \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": rpc error: code = NotFound desc = could not find container \"a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821\": container with ID starting with a23a90c2e70822b4824a0b6c710da34a6e3d179dbef5f87e89a4a625ba9b6821 not found: ID does not exist" Oct 30 20:51:13.759122 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759053 4988 scope.go:122] "RemoveContainer" containerID="804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab" Oct 30 20:51:13.759256 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759238 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab"} err="failed to get container status \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": rpc error: code = NotFound desc = could not find container \"804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab\": container with ID starting with 804dde41715e4a575d82dfe1ebddeaa2c0288ba84cb7d27f0ab96f7d9bc730ab not found: ID does not exist" Oct 30 20:51:13.759305 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759257 4988 scope.go:122] "RemoveContainer" containerID="c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9" Oct 30 20:51:13.759464 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759446 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9"} err="failed to get container status \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": rpc error: code = NotFound desc = could not find container \"c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9\": container with ID starting with c8b452192d61b5d4c65e0da2473e994763595d59eeabe59234d98db71b9c74b9 not found: ID does not exist" Oct 30 20:51:13.759535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759464 4988 scope.go:122] "RemoveContainer" containerID="f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47" Oct 30 20:51:13.759661 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.759643 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47"} err="failed to get container status \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": rpc error: code = NotFound desc = could not find container \"f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47\": container with ID starting with f582b213ea6812db6ab2bd51333a0ba571e597344c554bd8a5a0b7ac87560d47 not found: ID does not exist" Oct 30 20:51:13.780609 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780590 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.780694 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780619 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.780694 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780661 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.780694 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780686 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.780862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780712 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.780862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780729 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.780862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.780770 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db\") pod \"ee07f7c6-92e0-4599-842a-eec061e7439e\" (UID: \"ee07f7c6-92e0-4599-842a-eec061e7439e\") " Oct 30 20:51:13.781262 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.781238 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:13.782930 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.782908 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.783058 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.783032 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "kube-api-access-cj426". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:13.783123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.783106 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:13.783201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.783170 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:51:13.786755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.786729 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.794763 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.794742 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config" pod "ee07f7c6-92e0-4599-842a-eec061e7439e" (UID: "ee07f7c6-92e0-4599-842a-eec061e7439e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:13.881741 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881716 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.881741 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881736 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.881874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881746 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj426\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-kube-api-access-cj426\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.881874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881756 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee07f7c6-92e0-4599-842a-eec061e7439e-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.881874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881766 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.881874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881774 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee07f7c6-92e0-4599-842a-eec061e7439e-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:13.881874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:13.881783 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ee07f7c6-92e0-4599-842a-eec061e7439e-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:14.006482 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.006454 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:51:14.015924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.015898 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b333c4-fbd6-42ae-8718-80664f887846" path="/var/lib/kubelet/pods/c8b333c4-fbd6-42ae-8718-80664f887846/volumes" Oct 30 20:51:14.016314 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.016292 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" path="/var/lib/kubelet/pods/ff1ffa1d-6902-4428-9cfe-c4948086419d/volumes" Oct 30 20:51:14.016805 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.016789 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:51:14.723608 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.723583 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:51:14.724034 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.723577 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ee07f7c6-92e0-4599-842a-eec061e7439e","Type":"ContainerDied","Data":"9bb397d51ab2551fa900f5a93f9e2ce81a1c5259f6e36552865d18afd6e7cd52"} Oct 30 20:51:14.724034 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.723717 4988 scope.go:122] "RemoveContainer" containerID="13d87c59dc329c04e4fc286cf36708252552e6fe87b91343d3bc02ce3fbb58b9" Oct 30 20:51:14.731574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.731554 4988 scope.go:122] "RemoveContainer" containerID="b1fb6d388761f7432f1ff0b8f3504b805cbbd3a3835cfc9d9d2a416fdcad9595" Oct 30 20:51:14.738657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.738634 4988 scope.go:122] "RemoveContainer" containerID="ec3ca33346b7bc675ddc2ca09222c528bfc097510b5325a8ada0baff45c562f2" Oct 30 20:51:14.890244 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.890216 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:51:14.939814 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:14.939777 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:51:15.121006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.120966 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:51:15.121221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.121194 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c4b931a9-06ed-4af3-86c7-0cee62477555" containerName="otc-container" containerID="cri-o://4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99" gracePeriod=30 Oct 30 20:51:15.178437 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.178339 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:15.334079 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.334042 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:15.359540 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.359519 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:15.393892 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.393867 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85\" (UniqueName: \"kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85\") pod \"c4b931a9-06ed-4af3-86c7-0cee62477555\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " Oct 30 20:51:15.393996 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.393908 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs\") pod \"c4b931a9-06ed-4af3-86c7-0cee62477555\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " Oct 30 20:51:15.393996 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.393942 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert\") pod \"c4b931a9-06ed-4af3-86c7-0cee62477555\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " Oct 30 20:51:15.393996 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.393978 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs\") pod \"c4b931a9-06ed-4af3-86c7-0cee62477555\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " Oct 30 20:51:15.394159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.394038 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal\") pod \"c4b931a9-06ed-4af3-86c7-0cee62477555\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " Oct 30 20:51:15.394159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.394098 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs\") pod \"c4b931a9-06ed-4af3-86c7-0cee62477555\" (UID: \"c4b931a9-06ed-4af3-86c7-0cee62477555\") " Oct 30 20:51:15.394519 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.394478 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal" pod "c4b931a9-06ed-4af3-86c7-0cee62477555" (UID: "c4b931a9-06ed-4af3-86c7-0cee62477555"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:15.395983 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.395956 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85" pod "c4b931a9-06ed-4af3-86c7-0cee62477555" (UID: "c4b931a9-06ed-4af3-86c7-0cee62477555"). InnerVolumeSpecName "kube-api-access-jwb85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:15.396467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.396426 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs" pod "c4b931a9-06ed-4af3-86c7-0cee62477555" (UID: "c4b931a9-06ed-4af3-86c7-0cee62477555"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:15.396467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.396447 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs" pod "c4b931a9-06ed-4af3-86c7-0cee62477555" (UID: "c4b931a9-06ed-4af3-86c7-0cee62477555"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:15.396557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.396503 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs" pod "c4b931a9-06ed-4af3-86c7-0cee62477555" (UID: "c4b931a9-06ed-4af3-86c7-0cee62477555"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:15.396557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.396518 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert" pod "c4b931a9-06ed-4af3-86c7-0cee62477555" (UID: "c4b931a9-06ed-4af3-86c7-0cee62477555"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:15.494874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.494812 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c4b931a9-06ed-4af3-86c7-0cee62477555-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:15.494874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.494838 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:15.494874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.494852 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwb85\" (UniqueName: \"kubernetes.io/projected/c4b931a9-06ed-4af3-86c7-0cee62477555-kube-api-access-jwb85\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:15.494874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.494865 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:15.495062 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.494880 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:15.495062 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.494893 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b931a9-06ed-4af3-86c7-0cee62477555-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:51:15.629434 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.629383 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:15.730325 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.730291 4988 generic.go:356] "Generic (PLEG): container finished" podID="c4b931a9-06ed-4af3-86c7-0cee62477555" containerID="4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99" exitCode=0 Oct 30 20:51:15.730730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.730336 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:15.730730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.730371 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c4b931a9-06ed-4af3-86c7-0cee62477555","Type":"ContainerDied","Data":"4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99"} Oct 30 20:51:15.730730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.730431 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c4b931a9-06ed-4af3-86c7-0cee62477555","Type":"ContainerDied","Data":"626d5f3a6f6fe9e7614b729144c033209ec0418666a4c219fbfea6fd5db7d3f7"} Oct 30 20:51:15.730730 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.730458 4988 scope.go:122] "RemoveContainer" containerID="4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99" Oct 30 20:51:15.738672 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.738656 4988 scope.go:122] "RemoveContainer" containerID="4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99" Oct 30 20:51:15.738912 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:15.738894 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99\": container with ID starting with 4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99 not found: ID does not exist" containerID="4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99" Oct 30 20:51:15.738977 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.738921 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99"} err="failed to get container status \"4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99\": rpc error: code = NotFound desc = could not find container \"4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99\": container with ID starting with 4b0dce19eb7dd8b002558bd2f78c7afa058d072199755459ae3d8e1bdbb53b99 not found: ID does not exist" Oct 30 20:51:15.894203 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.894177 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:51:15.899100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:15.899079 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:51:16.015019 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:16.014989 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" path="/var/lib/kubelet/pods/8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d/volumes" Oct 30 20:51:16.015482 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:16.015468 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b931a9-06ed-4af3-86c7-0cee62477555" path="/var/lib/kubelet/pods/c4b931a9-06ed-4af3-86c7-0cee62477555/volumes" Oct 30 20:51:16.016068 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:16.016055 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" path="/var/lib/kubelet/pods/ee07f7c6-92e0-4599-842a-eec061e7439e/volumes" Oct 30 20:51:16.803185 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:16.803151 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:16.996943 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:16.996906 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:17.156255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:17.156222 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:17.306447 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:17.306401 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:23.223947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:23.223913 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:23.379203 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:23.379171 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:23.538509 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:23.538440 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:23.682177 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:23.682153 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:25.460961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.460930 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:25.487153 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487119 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:51:25.487515 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487492 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="alertmanager" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487517 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="alertmanager" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487536 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="init-config-reloader" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487544 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="init-config-reloader" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487561 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c8b333c4-fbd6-42ae-8718-80664f887846" containerName="thanos-querier" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487568 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b333c4-fbd6-42ae-8718-80664f887846" containerName="thanos-querier" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487582 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="method-gate" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487589 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="method-gate" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487604 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="config-reloader" Oct 30 20:51:25.487620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487610 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487624 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="kube-rbac-proxy" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487632 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="kube-rbac-proxy" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487641 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c4b931a9-06ed-4af3-86c7-0cee62477555" containerName="otc-container" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487649 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b931a9-06ed-4af3-86c7-0cee62477555" containerName="otc-container" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487661 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="prom-label-proxy" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487668 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="prom-label-proxy" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487678 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="init-config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487685 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="init-config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487698 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="thanos-sidecar" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487705 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="thanos-sidecar" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487724 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="prometheus" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487731 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="prometheus" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487748 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487755 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487842 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="prom-label-proxy" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487855 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="thanos-sidecar" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487865 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="prometheus" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487875 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7d2ea7-6dfe-41b7-8692-d6e9ec21727d" containerName="config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487886 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="alertmanager" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487894 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b333c4-fbd6-42ae-8718-80664f887846" containerName="thanos-querier" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487903 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="method-gate" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487913 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee07f7c6-92e0-4599-842a-eec061e7439e" containerName="config-reloader" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487922 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1ffa1d-6902-4428-9cfe-c4948086419d" containerName="kube-rbac-proxy" Oct 30 20:51:25.488047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.487930 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b931a9-06ed-4af3-86c7-0cee62477555" containerName="otc-container" Oct 30 20:51:25.492566 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.492547 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.558143 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.558111 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:51:25.567239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.567212 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.567372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.567250 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.567372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.567280 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4ntf\" (UniqueName: \"kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.567473 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.567445 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.605885 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.605852 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:51:25.606225 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.606197 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 20:51:25.606340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.606226 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:51:25.606340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.606238 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:51:25.606445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.606387 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:51:25.606445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.606399 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-h8bkr\"" Oct 30 20:51:25.665646 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.665617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:25.668622 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.668596 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.668734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.668634 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.668734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.668662 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4ntf\" (UniqueName: \"kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.668734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.668704 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.669368 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.669346 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.670998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.670968 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.671099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.671011 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.733699 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.733632 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4ntf\" (UniqueName: \"kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf\") pod \"data-science-collector-collector-1\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.805561 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.805525 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:51:25.850243 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:25.850200 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:26.181128 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:26.181099 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:51:26.184471 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:51:26.184447 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5184f9_4fac_4546_aef1_0797528bcea2.slice/crio-d938879af67efbd2c9e6c3cc797a1d12e858e4fdb48b461f480ff07d9aa0315e WatchSource:0}: Error finding container d938879af67efbd2c9e6c3cc797a1d12e858e4fdb48b461f480ff07d9aa0315e: Status 404 returned error can't find the container with id d938879af67efbd2c9e6c3cc797a1d12e858e4fdb48b461f480ff07d9aa0315e Oct 30 20:51:26.186479 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:26.186443 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:51:26.186554 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:26.186514 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:51:26.392792 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:26.392756 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:51:26.778572 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:26.778542 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b5184f9-4fac-4546-aef1-0797528bcea2","Type":"ContainerStarted","Data":"8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1"} Oct 30 20:51:26.778924 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:26.778589 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b5184f9-4fac-4546-aef1-0797528bcea2","Type":"ContainerStarted","Data":"d938879af67efbd2c9e6c3cc797a1d12e858e4fdb48b461f480ff07d9aa0315e"} Oct 30 20:51:26.907901 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:26.907849 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.90783488 podStartE2EDuration="1.90783488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:26.90644481 +0000 UTC m=+915.013940711" watchObservedRunningTime="2026-10-30 20:51:26.90783488 +0000 UTC m=+915.015330854" Oct 30 20:51:27.029686 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.029604 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:51:27.032614 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.032596 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.092637 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.092616 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:51:27.092985 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.092972 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 20:51:27.093079 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.093063 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 20:51:27.093585 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.093570 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 20:51:27.093687 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.093673 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 20:51:27.093744 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.093681 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:51:27.093793 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.093744 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 20:51:27.094006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.093992 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-2qvqh\"" Oct 30 20:51:27.142622 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.142598 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:51:27.164083 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.164040 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:27.280991 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.280922 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.280991 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.280967 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.281144 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.281042 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.281144 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.281079 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f86b\" (UniqueName: \"kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.281144 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.281112 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.281263 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.281196 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.281263 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.281235 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.281334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.281266 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382088 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382064 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382199 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382094 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382199 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382121 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382199 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382142 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382199 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382172 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382378 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382203 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382378 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382228 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f86b\" (UniqueName: \"kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382378 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382269 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.382959 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382933 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.383069 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.382967 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.383069 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.383000 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.383465 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.383442 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.384598 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.384571 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.384703 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.384685 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.384744 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.384729 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.442951 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.442930 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f86b\" (UniqueName: \"kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:27.443840 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:27.443818 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:35.341479 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.341451 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:35.446334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.446305 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:35.543072 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.543044 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:35.543196 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.543145 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:35.595125 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.595074 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:35.733067 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.733043 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:35.796686 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:35.796664 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.061737 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.061712 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.578439 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:51:36.578394 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77492087_e84d_4d4d_9b1b_0dac78c66076.slice/crio-7af79ecc072f34d6282b977bdc9ca0969426525b1e110a6e4eaf16b318900fef WatchSource:0}: Error finding container 7af79ecc072f34d6282b977bdc9ca0969426525b1e110a6e4eaf16b318900fef: Status 404 returned error can't find the container with id 7af79ecc072f34d6282b977bdc9ca0969426525b1e110a6e4eaf16b318900fef Oct 30 20:51:36.590625 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.590596 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:51:36.820523 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820493 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerStarted","Data":"b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b"} Oct 30 20:51:36.820657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820545 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerStarted","Data":"9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d"} Oct 30 20:51:36.820657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820568 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerStarted","Data":"9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e"} Oct 30 20:51:36.820657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820613 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.820799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820662 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.820799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820690 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.820799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.820712 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerStarted","Data":"7af79ecc072f34d6282b977bdc9ca0969426525b1e110a6e4eaf16b318900fef"} Oct 30 20:51:36.822980 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.822961 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.829587 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.829537 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:51:36.943206 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:36.943170 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=10.943159755 podStartE2EDuration="10.943159755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:36.942660546 +0000 UTC m=+925.050156459" watchObservedRunningTime="2026-10-30 20:51:36.943159755 +0000 UTC m=+925.050655662" Oct 30 20:51:56.393734 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:51:56.393657 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:51:58.411615 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:58.411571 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:58.832436 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:51:58.832328 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:52:06.912092 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:06.912049 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:08.832213 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:08.832167 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:52:10.477178 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:10.477140 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:18.831931 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:18.831896 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:24.021213 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.021178 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:52:24.021722 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.021528 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="3b5184f9-4fac-4546-aef1-0797528bcea2" containerName="otc-container" containerID="cri-o://8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1" gracePeriod=30 Oct 30 20:52:24.090531 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.090491 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:24.269602 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.269577 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:24.310935 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.310857 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:24.352881 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.352848 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs\") pod \"3b5184f9-4fac-4546-aef1-0797528bcea2\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " Oct 30 20:52:24.353076 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.352893 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf\" (UniqueName: \"kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf\") pod \"3b5184f9-4fac-4546-aef1-0797528bcea2\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " Oct 30 20:52:24.353076 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.352914 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal\") pod \"3b5184f9-4fac-4546-aef1-0797528bcea2\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " Oct 30 20:52:24.353076 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.353004 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs\") pod \"3b5184f9-4fac-4546-aef1-0797528bcea2\" (UID: \"3b5184f9-4fac-4546-aef1-0797528bcea2\") " Oct 30 20:52:24.353629 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.353581 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal" pod "3b5184f9-4fac-4546-aef1-0797528bcea2" (UID: "3b5184f9-4fac-4546-aef1-0797528bcea2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:24.355738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.355698 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf" pod "3b5184f9-4fac-4546-aef1-0797528bcea2" (UID: "3b5184f9-4fac-4546-aef1-0797528bcea2"). InnerVolumeSpecName "kube-api-access-g4ntf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:24.356226 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.356198 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs" pod "3b5184f9-4fac-4546-aef1-0797528bcea2" (UID: "3b5184f9-4fac-4546-aef1-0797528bcea2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:24.356344 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.356233 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs" pod "3b5184f9-4fac-4546-aef1-0797528bcea2" (UID: "3b5184f9-4fac-4546-aef1-0797528bcea2"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:24.454235 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.454205 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:24.454235 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.454232 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5184f9-4fac-4546-aef1-0797528bcea2-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:24.454447 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.454242 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4ntf\" (UniqueName: \"kubernetes.io/projected/3b5184f9-4fac-4546-aef1-0797528bcea2-kube-api-access-g4ntf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:24.454447 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.454252 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b5184f9-4fac-4546-aef1-0797528bcea2-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:24.510813 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.510769 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:24.780097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.780062 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:52:24.780344 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.780290 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo" containerID="cri-o://9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e" gracePeriod=30 Oct 30 20:52:24.780479 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.780341 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway" containerID="cri-o://9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d" gracePeriod=30 Oct 30 20:52:24.780479 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.780315 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway-opa" containerID="cri-o://b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b" gracePeriod=30 Oct 30 20:52:24.821303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:24.821270 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:25.034820 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.034743 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:25.040462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.040433 4988 generic.go:356] "Generic (PLEG): container finished" podID="3b5184f9-4fac-4546-aef1-0797528bcea2" containerID="8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1" exitCode=0 Oct 30 20:52:25.040635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.040485 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:25.040635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.040516 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b5184f9-4fac-4546-aef1-0797528bcea2","Type":"ContainerDied","Data":"8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1"} Oct 30 20:52:25.040635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.040574 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b5184f9-4fac-4546-aef1-0797528bcea2","Type":"ContainerDied","Data":"d938879af67efbd2c9e6c3cc797a1d12e858e4fdb48b461f480ff07d9aa0315e"} Oct 30 20:52:25.040635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.040603 4988 scope.go:122] "RemoveContainer" containerID="8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1" Oct 30 20:52:25.044773 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.044745 4988 generic.go:356] "Generic (PLEG): container finished" podID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerID="b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b" exitCode=0 Oct 30 20:52:25.044903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.044774 4988 generic.go:356] "Generic (PLEG): container finished" podID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerID="9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e" exitCode=0 Oct 30 20:52:25.044903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.044805 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerDied","Data":"b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b"} Oct 30 20:52:25.044903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.044856 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerDied","Data":"9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e"} Oct 30 20:52:25.051183 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.051153 4988 scope.go:122] "RemoveContainer" containerID="8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1" Oct 30 20:52:25.051600 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:52:25.051574 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1\": container with ID starting with 8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1 not found: ID does not exist" containerID="8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1" Oct 30 20:52:25.051693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.051610 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1"} err="failed to get container status \"8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1\": rpc error: code = NotFound desc = could not find container \"8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1\": container with ID starting with 8d27160fcc882333159a11f2afcd33941ce4b681c7f7bdcd6ab676a8afcc62a1 not found: ID does not exist" Oct 30 20:52:25.182837 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.182789 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:25.206353 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.206317 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:52:25.211077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:25.211050 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:52:26.015802 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.015768 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5184f9-4fac-4546-aef1-0797528bcea2" path="/var/lib/kubelet/pods/3b5184f9-4fac-4546-aef1-0797528bcea2/volumes" Oct 30 20:52:26.036144 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.036120 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:26.051395 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.051359 4988 generic.go:356] "Generic (PLEG): container finished" podID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerID="9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d" exitCode=0 Oct 30 20:52:26.051596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.051436 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:26.051596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.051444 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerDied","Data":"9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d"} Oct 30 20:52:26.051596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.051498 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"77492087-e84d-4d4d-9b1b-0dac78c66076","Type":"ContainerDied","Data":"7af79ecc072f34d6282b977bdc9ca0969426525b1e110a6e4eaf16b318900fef"} Oct 30 20:52:26.051596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.051532 4988 scope.go:122] "RemoveContainer" containerID="b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b" Oct 30 20:52:26.061269 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.061250 4988 scope.go:122] "RemoveContainer" containerID="9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d" Oct 30 20:52:26.068387 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068365 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068516 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068400 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068541 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b\" (UniqueName: \"kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068656 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068641 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068715 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068668 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068772 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068715 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068772 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068749 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068779 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.068870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.068830 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf\") pod \"77492087-e84d-4d4d-9b1b-0dac78c66076\" (UID: \"77492087-e84d-4d4d-9b1b-0dac78c66076\") " Oct 30 20:52:26.069214 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.069135 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:26.069214 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.069162 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:26.069379 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.069353 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:26.069529 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.069401 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:26.070182 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.070060 4988 scope.go:122] "RemoveContainer" containerID="9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e" Oct 30 20:52:26.071291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.071258 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "kube-api-access-9f86b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:26.071662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.071639 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:52:26.071760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.071710 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:26.071760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.071733 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:26.072511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.072492 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "77492087-e84d-4d4d-9b1b-0dac78c66076" (UID: "77492087-e84d-4d4d-9b1b-0dac78c66076"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:26.087885 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.087865 4988 scope.go:122] "RemoveContainer" containerID="b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b" Oct 30 20:52:26.088210 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:52:26.088187 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b\": container with ID starting with b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b not found: ID does not exist" containerID="b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b" Oct 30 20:52:26.088309 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.088222 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b"} err="failed to get container status \"b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b\": rpc error: code = NotFound desc = could not find container \"b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b\": container with ID starting with b0428055a15b492a615731e60b5ae1976603b97c82ec68b909ba59abeab1845b not found: ID does not exist" Oct 30 20:52:26.088309 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.088245 4988 scope.go:122] "RemoveContainer" containerID="9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d" Oct 30 20:52:26.088535 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:52:26.088511 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d\": container with ID starting with 9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d not found: ID does not exist" containerID="9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d" Oct 30 20:52:26.088627 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.088544 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d"} err="failed to get container status \"9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d\": rpc error: code = NotFound desc = could not find container \"9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d\": container with ID starting with 9af6712a5e20322da0b5880fa06d39c8668496b9ae09b29a7795fdf1db82d42d not found: ID does not exist" Oct 30 20:52:26.088627 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.088568 4988 scope.go:122] "RemoveContainer" containerID="9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e" Oct 30 20:52:26.088896 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:52:26.088862 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e\": container with ID starting with 9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e not found: ID does not exist" containerID="9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e" Oct 30 20:52:26.088958 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.088906 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e"} err="failed to get container status \"9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e\": rpc error: code = NotFound desc = could not find container \"9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e\": container with ID starting with 9b077d5303afde4af958c2e908f593c5ddaaca143e5df605ddaeb7cca809745e not found: ID does not exist" Oct 30 20:52:26.169483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169453 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169480 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169494 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f86b\" (UniqueName: \"kubernetes.io/projected/77492087-e84d-4d4d-9b1b-0dac78c66076-kube-api-access-9f86b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169527 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169540 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169552 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169565 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169579 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.169693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.169591 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/77492087-e84d-4d4d-9b1b-0dac78c66076-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.181244 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.181223 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:52:26.270589 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.270553 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 20:52:26.393270 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:52:26.393236 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:52:26.473969 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.473920 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:52:26.476977 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:26.476953 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:52:28.015620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:28.015588 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" path="/var/lib/kubelet/pods/77492087-e84d-4d4d-9b1b-0dac78c66076/volumes" Oct 30 20:52:38.588892 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:38.588856 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:38.742425 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:38.742375 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.012555 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.012516 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.181018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.180979 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.338600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.338523 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.496180 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.496148 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.656008 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.655973 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5"] Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656273 4988 cpu_manager.go:414] "containerMap: removing container" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656284 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656294 4988 cpu_manager.go:414] "containerMap: removing container" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway-opa" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656299 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway-opa" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656311 4988 cpu_manager.go:414] "containerMap: removing container" podUID="3b5184f9-4fac-4546-aef1-0797528bcea2" containerName="otc-container" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656317 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5184f9-4fac-4546-aef1-0797528bcea2" containerName="otc-container" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656334 4988 cpu_manager.go:414] "containerMap: removing container" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656340 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway" Oct 30 20:52:39.656392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656389 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5184f9-4fac-4546-aef1-0797528bcea2" containerName="otc-container" Oct 30 20:52:39.656690 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656399 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway" Oct 30 20:52:39.656690 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656420 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo" Oct 30 20:52:39.656690 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.656425 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="77492087-e84d-4d4d-9b1b-0dac78c66076" containerName="tempo-gateway-opa" Oct 30 20:52:39.659681 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.659665 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.674531 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.674496 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.712159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.712127 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-mh276\"" Oct 30 20:52:39.761880 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.761850 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:52:39.762082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.761848 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 20:52:39.762082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.761909 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 20:52:39.762082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.761884 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 20:52:39.762082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.761914 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:52:39.762314 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.762082 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 20:52:39.763323 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.763303 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5"] Oct 30 20:52:39.777188 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.777162 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr"] Oct 30 20:52:39.780815 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.780799 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.784648 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.784624 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.784763 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.784660 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.784763 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.784690 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.784864 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.784797 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.784864 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.784854 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkqs\" (UniqueName: \"kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.834733 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.834703 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-7qgpd\"" Oct 30 20:52:39.834917 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.834765 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 20:52:39.834917 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.834783 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 20:52:39.840311 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.840284 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr"] Oct 30 20:52:39.885469 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.885434 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.885657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.885481 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.885657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.885507 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.885657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.885547 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.885657 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.885582 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.885824 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.885716 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkqs\" (UniqueName: \"kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.886291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.886262 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.886291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.886266 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.886553 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.886363 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsn5f\" (UniqueName: \"kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.886648 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.886631 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.886865 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.886849 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.888179 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.888154 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.910885 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.910809 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.960739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.960708 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkqs\" (UniqueName: \"kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.976530 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.976490 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:52:39.986995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.986969 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.987104 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.987006 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.987104 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.987039 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsn5f\" (UniqueName: \"kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.987586 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.987562 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:39.989325 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:39.989302 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:40.048285 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:40.048248 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsn5f\" (UniqueName: \"kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f\") pod \"data-science-thanos-querier-proxy-77657866bd-dvxcr\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:40.096988 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:40.096945 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:52:40.356462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:40.356248 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5"] Oct 30 20:52:40.359074 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:52:40.359044 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c3d99_db81_411b_8701_5974fcbcc399.slice/crio-5cda68e045dba4718b93b8adc857761faae9ae84b4dd9723804f8781b27f396e WatchSource:0}: Error finding container 5cda68e045dba4718b93b8adc857761faae9ae84b4dd9723804f8781b27f396e: Status 404 returned error can't find the container with id 5cda68e045dba4718b93b8adc857761faae9ae84b4dd9723804f8781b27f396e Oct 30 20:52:40.471485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:40.471458 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr"] Oct 30 20:52:40.475000 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:52:40.474975 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1a7232_073a_47c4_8b5e_a4ddab7f7589.slice/crio-a7303c9402b950612265ae64683106ba318490bf6146e24750672c90a74e445f WatchSource:0}: Error finding container a7303c9402b950612265ae64683106ba318490bf6146e24750672c90a74e445f: Status 404 returned error can't find the container with id a7303c9402b950612265ae64683106ba318490bf6146e24750672c90a74e445f Oct 30 20:52:41.115338 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.115297 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerStarted","Data":"49a5972451ab2e5682a6cb2cd3c7d03466181d75da6fd057637976264e409c5e"} Oct 30 20:52:41.115859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.115378 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerStarted","Data":"33afa49a3583a04aada279a2fc06e10f4f3d1343553f26cccb5481f7c0174281"} Oct 30 20:52:41.115859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.115429 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerStarted","Data":"0cb3846c5e64291b4cd5596f0f0dd908956e047c924db94a31e17a059134d7a0"} Oct 30 20:52:41.115859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.115456 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerStarted","Data":"a7303c9402b950612265ae64683106ba318490bf6146e24750672c90a74e445f"} Oct 30 20:52:41.117270 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.117246 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerStarted","Data":"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f"} Oct 30 20:52:41.117381 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.117295 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerStarted","Data":"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97"} Oct 30 20:52:41.117381 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.117318 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerStarted","Data":"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc"} Oct 30 20:52:41.117381 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.117344 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerStarted","Data":"5cda68e045dba4718b93b8adc857761faae9ae84b4dd9723804f8781b27f396e"} Oct 30 20:52:41.234915 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.234862 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" podStartSLOduration=2.234848193 podStartE2EDuration="2.234848193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:41.233040064 +0000 UTC m=+989.340535973" watchObservedRunningTime="2026-10-30 20:52:41.234848193 +0000 UTC m=+989.342344103" Oct 30 20:52:41.263492 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.263455 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:41.366214 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:41.366115 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" podStartSLOduration=2.366100911 podStartE2EDuration="2.366100911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:41.365492021 +0000 UTC m=+989.472987932" watchObservedRunningTime="2026-10-30 20:52:41.366100911 +0000 UTC m=+989.473596821" Oct 30 20:52:44.748398 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.748361 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:44.769951 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.769915 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:52:44.773325 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.773306 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.840318 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.840286 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:52:44.847988 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.847956 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:52:44.878596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.878564 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:52:44.878762 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.878700 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-vtvlx\"" Oct 30 20:52:44.878762 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.878721 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:52:44.887535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.887499 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:52:44.887760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.887740 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 20:52:44.932546 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.932509 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.932716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.932557 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.932716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.932587 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2rj8\" (UniqueName: \"kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.932716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.932649 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.932716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.932683 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.932716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.932713 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:44.937206 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:44.937175 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:45.033303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.033216 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.033303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.033262 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.033543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.033316 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.033543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.033347 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.033543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.033374 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2rj8\" (UniqueName: \"kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.033543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.033466 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.034127 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.034102 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.035866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.035842 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.036130 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.036112 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.036185 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.036130 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.036380 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.036364 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.102462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.102397 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2rj8\" (UniqueName: \"kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8\") pod \"data-science-collector-collector-1\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.122969 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.122929 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:45.324558 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.324464 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:45.386698 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.386668 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:52:45.766249 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.766216 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:52:45.769081 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:52:45.769053 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa82a77c_2bab_49ee_a17b_49b56f8ff243.slice/crio-2f8a1af7701fdf2c57c8d457c13e0bc142db6cec7b5b7e5254add37619938948 WatchSource:0}: Error finding container 2f8a1af7701fdf2c57c8d457c13e0bc142db6cec7b5b7e5254add37619938948: Status 404 returned error can't find the container with id 2f8a1af7701fdf2c57c8d457c13e0bc142db6cec7b5b7e5254add37619938948 Oct 30 20:52:45.771117 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.771081 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:52:45.771194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:45.771142 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:52:46.140165 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:46.140135 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fa82a77c-2bab-49ee-a17b-49b56f8ff243","Type":"ContainerStarted","Data":"1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e"} Oct 30 20:52:46.140362 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:46.140196 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fa82a77c-2bab-49ee-a17b-49b56f8ff243","Type":"ContainerStarted","Data":"2f8a1af7701fdf2c57c8d457c13e0bc142db6cec7b5b7e5254add37619938948"} Oct 30 20:52:46.262292 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:46.262233 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.2622165770000002 podStartE2EDuration="2.262216577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:46.260948578 +0000 UTC m=+994.368444488" watchObservedRunningTime="2026-10-30 20:52:46.262216577 +0000 UTC m=+994.369712487" Oct 30 20:52:46.641496 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:46.641458 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:46.980935 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:46.980849 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:52:46.984805 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:46.984781 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.014605 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.014572 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:47.039114 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.039086 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:52:47.039287 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.039184 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 20:52:47.088351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.088318 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 20:52:47.088351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.088328 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:52:47.088351 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.088328 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 20:52:47.088639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.088371 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 20:52:47.088639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.088371 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-dhtwc\"" Oct 30 20:52:47.088639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.088495 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 20:52:47.089867 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.089851 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:52:47.242522 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.242436 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:47.252512 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.252483 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8x5\" (UniqueName: \"kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.252760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.252732 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.252873 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.252851 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.253053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.253031 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.253168 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.253147 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.253230 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.253202 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.253281 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.253251 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.253333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.253290 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354113 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8x5\" (UniqueName: \"kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354156 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354203 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354240 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354270 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354309 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354330 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.354672 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.354352 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.355086 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.355056 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.355281 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.355129 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.355367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.355196 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.355367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.355197 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.356795 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.356773 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.356861 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.356823 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.356908 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.356885 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:47.421109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:47.421073 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8x5\" (UniqueName: \"kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:56.393028 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:52:56.392988 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:52:56.632471 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:56.632385 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:56.738429 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:56.738328 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:56.834192 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:56.834141 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:56.834402 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:56.834268 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:56.886467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:56.886435 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:57.033651 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:57.033621 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:57.098701 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:57.098674 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:57.200618 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:57.200580 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:57.664591 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:57.664557 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:57.686036 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:57.686008 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:52:57.689288 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:52:57.689265 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af6e281_bed5_48e3_a1c3_c0d75c4d5e57.slice/crio-9e4cca948fc4d138fe6e3b631ced40a4743e851dded0ede996a6d3bb7908b957 WatchSource:0}: Error finding container 9e4cca948fc4d138fe6e3b631ced40a4743e851dded0ede996a6d3bb7908b957: Status 404 returned error can't find the container with id 9e4cca948fc4d138fe6e3b631ced40a4743e851dded0ede996a6d3bb7908b957 Oct 30 20:52:58.023540 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.023463 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:58.195512 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195465 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerStarted","Data":"5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd"} Oct 30 20:52:58.195714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195538 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerStarted","Data":"84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20"} Oct 30 20:52:58.195714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195677 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:58.195826 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195722 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerStarted","Data":"acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe"} Oct 30 20:52:58.195826 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195751 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerStarted","Data":"9e4cca948fc4d138fe6e3b631ced40a4743e851dded0ede996a6d3bb7908b957"} Oct 30 20:52:58.195937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195875 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:58.195937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.195919 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:58.198705 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.198683 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:58.202100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.202071 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:58.205324 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.205303 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:52:58.379478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.379379 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:58.449075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:58.449016 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=12.448999877 podStartE2EDuration="12.448999877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:58.324148114 +0000 UTC m=+1006.431644025" watchObservedRunningTime="2026-10-30 20:52:58.448999877 +0000 UTC m=+1006.556495789" Oct 30 20:52:59.018677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.018627 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:59.190610 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.190579 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:52:59.192961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.192925 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:59.194940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.194920 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.252461 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.252429 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 20:52:59.256182 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.256162 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.303405 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.303310 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:52:59.316732 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.316699 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 20:52:59.316904 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.316782 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 20:52:59.316904 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.316819 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 20:52:59.350542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350483 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 20:52:59.350542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350484 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 20:52:59.350542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350485 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 20:52:59.350542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350483 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350564 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-g6pst\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350668 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350674 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350703 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350730 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350750 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-t5vs6\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350754 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 20:52:59.350860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.350861 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 20:52:59.352074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.352054 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 20:52:59.457556 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457523 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457556 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457559 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457577 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457633 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457685 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457708 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457732 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457831 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457854 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457877 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvvm\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.457987 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457960 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.458021 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.457983 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.559466 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559372 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.559466 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559438 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.559644 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559545 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrg8\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.559644 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559596 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.559710 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559663 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.559776 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559758 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.559905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559891 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.559944 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559937 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.559989 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.559958 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560036 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560014 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560079 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560054 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560130 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560091 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.560130 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560119 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560146 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560192 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560253 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.560340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560304 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560470 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560368 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560470 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560392 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvvm\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.560575 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.560466 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.561129 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.561097 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.561621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.561361 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.562842 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.562816 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.563066 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.563044 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.563151 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.563063 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.563310 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.563281 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.563757 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.563733 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.563868 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.563853 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.564186 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.564162 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.564291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.564271 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.627869 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.627828 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvvm\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:52:59.661746 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661706 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.661934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661763 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.661934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661792 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.661934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661822 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.661934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661841 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.661934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661866 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrg8\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.661934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.661897 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.662238 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.662135 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.664268 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.664236 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.664404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.664265 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.664510 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.664486 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.664640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.664626 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.664679 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.664634 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.723007 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.722973 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrg8\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:52:59.816265 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:52:59.816184 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:53:00.244277 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:00.244222 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:53:00.247061 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:53:00.247031 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4d83ee_d9b1_4059_9fdc_dbb627a62178.slice/crio-48132e325d476942124153cf4ff166483bd99b3d3ee20b84d21b9d72cea1a25d WatchSource:0}: Error finding container 48132e325d476942124153cf4ff166483bd99b3d3ee20b84d21b9d72cea1a25d: Status 404 returned error can't find the container with id 48132e325d476942124153cf4ff166483bd99b3d3ee20b84d21b9d72cea1a25d Oct 30 20:53:01.213495 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:01.213451 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerStarted","Data":"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6"} Oct 30 20:53:01.213682 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:01.213526 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerStarted","Data":"48132e325d476942124153cf4ff166483bd99b3d3ee20b84d21b9d72cea1a25d"} Oct 30 20:53:04.229074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:04.229031 4988 generic.go:356] "Generic (PLEG): container finished" podID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerID="2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6" exitCode=0 Oct 30 20:53:04.229074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:04.229072 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerDied","Data":"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6"} Oct 30 20:53:05.235610 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:05.235577 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerStarted","Data":"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6"} Oct 30 20:53:05.236032 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:05.235637 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerStarted","Data":"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df"} Oct 30 20:53:05.236032 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:05.235808 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:53:05.360796 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:05.360745 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.274609727 podStartE2EDuration="6.360732692s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.086122965s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:05.357983347 +0000 UTC m=+1013.465479257" watchObservedRunningTime="2026-10-30 20:53:05.360732692 +0000 UTC m=+1013.468228602" Oct 30 20:53:06.730480 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:06.730443 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:06.834929 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:06.834895 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:06.932688 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:06.932648 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:06.932858 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:06.932796 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:06.990348 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:06.990264 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:07.131444 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:07.131384 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:07.139463 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:07.139437 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:07.203809 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:07.203780 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:07.376340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:07.376246 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:07.866145 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:53:07.866109 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251d4f40_968e_4e98_827e_0ccc962d07b4.slice/crio-cb122b2099d411b6ff858be9744c3d6fefaed79d38040d0e98c0bf4e0c2c98d2 WatchSource:0}: Error finding container cb122b2099d411b6ff858be9744c3d6fefaed79d38040d0e98c0bf4e0c2c98d2: Status 404 returned error can't find the container with id cb122b2099d411b6ff858be9744c3d6fefaed79d38040d0e98c0bf4e0c2c98d2 Oct 30 20:53:07.913377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:07.913345 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 20:53:08.250208 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:08.250167 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerStarted","Data":"cb122b2099d411b6ff858be9744c3d6fefaed79d38040d0e98c0bf4e0c2c98d2"} Oct 30 20:53:09.256390 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:09.256355 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerStarted","Data":"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27"} Oct 30 20:53:09.264772 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:09.264731 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:11.241092 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:11.241060 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:53:12.271717 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:12.271681 4988 generic.go:356] "Generic (PLEG): container finished" podID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerID="3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27" exitCode=0 Oct 30 20:53:12.272136 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:12.271757 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerDied","Data":"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27"} Oct 30 20:53:13.279611 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:13.279575 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerStarted","Data":"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602"} Oct 30 20:53:13.280006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:13.279645 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerStarted","Data":"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900"} Oct 30 20:53:13.280006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:13.279670 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerStarted","Data":"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446"} Oct 30 20:53:13.459255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:13.459162 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=11.272539975 podStartE2EDuration="14.459146505s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.18660653s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:13.409384224 +0000 UTC m=+1021.516880133" watchObservedRunningTime="2026-10-30 20:53:13.459146505 +0000 UTC m=+1021.566642424" Oct 30 20:53:17.377393 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:17.377348 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:20.208676 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:20.208632 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:53:22.376452 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:22.376360 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:22.381934 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:22.381910 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:23.328953 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:23.328926 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:26.395366 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:26.395330 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:53:30.208738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:30.208695 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:53:40.208912 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:40.208872 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:53:43.063581 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.063537 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5"] Oct 30 20:53:43.064100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.063884 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="kube-rbac-proxy" containerID="cri-o://88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" gracePeriod=30 Oct 30 20:53:43.064100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.063917 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="prom-label-proxy" containerID="cri-o://0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" gracePeriod=30 Oct 30 20:53:43.064100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.063943 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="method-gate" containerID="cri-o://2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" gracePeriod=30 Oct 30 20:53:43.167520 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.167477 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr"] Oct 30 20:53:43.167732 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.167707 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="kube-rbac-proxy" containerID="cri-o://0cb3846c5e64291b4cd5596f0f0dd908956e047c924db94a31e17a059134d7a0" gracePeriod=30 Oct 30 20:53:43.167838 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.167732 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="prom-label-proxy" containerID="cri-o://49a5972451ab2e5682a6cb2cd3c7d03466181d75da6fd057637976264e409c5e" gracePeriod=30 Oct 30 20:53:43.167838 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.167759 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="method-gate" containerID="cri-o://33afa49a3583a04aada279a2fc06e10f4f3d1343553f26cccb5481f7c0174281" gracePeriod=30 Oct 30 20:53:43.400669 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.400638 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:53:43.421382 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.421352 4988 generic.go:356] "Generic (PLEG): container finished" podID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerID="49a5972451ab2e5682a6cb2cd3c7d03466181d75da6fd057637976264e409c5e" exitCode=0 Oct 30 20:53:43.421382 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.421377 4988 generic.go:356] "Generic (PLEG): container finished" podID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerID="33afa49a3583a04aada279a2fc06e10f4f3d1343553f26cccb5481f7c0174281" exitCode=0 Oct 30 20:53:43.421382 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.421387 4988 generic.go:356] "Generic (PLEG): container finished" podID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerID="0cb3846c5e64291b4cd5596f0f0dd908956e047c924db94a31e17a059134d7a0" exitCode=0 Oct 30 20:53:43.421662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.421447 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerDied","Data":"49a5972451ab2e5682a6cb2cd3c7d03466181d75da6fd057637976264e409c5e"} Oct 30 20:53:43.421662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.421506 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerDied","Data":"33afa49a3583a04aada279a2fc06e10f4f3d1343553f26cccb5481f7c0174281"} Oct 30 20:53:43.421662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.421542 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerDied","Data":"0cb3846c5e64291b4cd5596f0f0dd908956e047c924db94a31e17a059134d7a0"} Oct 30 20:53:43.431976 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.431947 4988 generic.go:356] "Generic (PLEG): container finished" podID="d48c3d99-db81-411b-8701-5974fcbcc399" containerID="0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" exitCode=0 Oct 30 20:53:43.431976 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.431969 4988 generic.go:356] "Generic (PLEG): container finished" podID="d48c3d99-db81-411b-8701-5974fcbcc399" containerID="2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" exitCode=0 Oct 30 20:53:43.431976 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.431976 4988 generic.go:356] "Generic (PLEG): container finished" podID="d48c3d99-db81-411b-8701-5974fcbcc399" containerID="88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" exitCode=0 Oct 30 20:53:43.432293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.432028 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" Oct 30 20:53:43.432293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.432022 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerDied","Data":"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f"} Oct 30 20:53:43.432293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.432164 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerDied","Data":"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97"} Oct 30 20:53:43.432293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.432203 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerDied","Data":"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc"} Oct 30 20:53:43.432293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.432232 4988 scope.go:122] "RemoveContainer" containerID="0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" Oct 30 20:53:43.432293 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.432236 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5" event={"ID":"d48c3d99-db81-411b-8701-5974fcbcc399","Type":"ContainerDied","Data":"5cda68e045dba4718b93b8adc857761faae9ae84b4dd9723804f8781b27f396e"} Oct 30 20:53:43.442835 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.442630 4988 scope.go:122] "RemoveContainer" containerID="2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" Oct 30 20:53:43.446899 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.446875 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:53:43.451296 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.451275 4988 scope.go:122] "RemoveContainer" containerID="88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" Oct 30 20:53:43.459358 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.459334 4988 scope.go:122] "RemoveContainer" containerID="0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" Oct 30 20:53:43.459683 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:43.459662 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": container with ID starting with 0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f not found: ID does not exist" containerID="0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" Oct 30 20:53:43.459773 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.459694 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f"} err="failed to get container status \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": rpc error: code = NotFound desc = could not find container \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": container with ID starting with 0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f not found: ID does not exist" Oct 30 20:53:43.459773 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.459712 4988 scope.go:122] "RemoveContainer" containerID="2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" Oct 30 20:53:43.459955 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:43.459936 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": container with ID starting with 2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97 not found: ID does not exist" containerID="2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" Oct 30 20:53:43.460013 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.459960 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97"} err="failed to get container status \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": rpc error: code = NotFound desc = could not find container \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": container with ID starting with 2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97 not found: ID does not exist" Oct 30 20:53:43.460013 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.459975 4988 scope.go:122] "RemoveContainer" containerID="88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" Oct 30 20:53:43.460238 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:43.460217 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": container with ID starting with 88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc not found: ID does not exist" containerID="88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" Oct 30 20:53:43.460306 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.460243 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc"} err="failed to get container status \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": rpc error: code = NotFound desc = could not find container \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": container with ID starting with 88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc not found: ID does not exist" Oct 30 20:53:43.460306 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.460258 4988 scope.go:122] "RemoveContainer" containerID="0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" Oct 30 20:53:43.460536 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.460515 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f"} err="failed to get container status \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": rpc error: code = NotFound desc = could not find container \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": container with ID starting with 0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f not found: ID does not exist" Oct 30 20:53:43.460536 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.460536 4988 scope.go:122] "RemoveContainer" containerID="2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" Oct 30 20:53:43.460767 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.460745 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97"} err="failed to get container status \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": rpc error: code = NotFound desc = could not find container \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": container with ID starting with 2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97 not found: ID does not exist" Oct 30 20:53:43.460767 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.460766 4988 scope.go:122] "RemoveContainer" containerID="88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" Oct 30 20:53:43.461041 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461022 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc"} err="failed to get container status \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": rpc error: code = NotFound desc = could not find container \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": container with ID starting with 88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc not found: ID does not exist" Oct 30 20:53:43.461120 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461043 4988 scope.go:122] "RemoveContainer" containerID="0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f" Oct 30 20:53:43.461279 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461263 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f"} err="failed to get container status \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": rpc error: code = NotFound desc = could not find container \"0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f\": container with ID starting with 0691a75f41d91ae85d4337bbc5b1042fa99f2d9c7aee4d4f7b6b1f5c47a20a6f not found: ID does not exist" Oct 30 20:53:43.461337 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461279 4988 scope.go:122] "RemoveContainer" containerID="2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97" Oct 30 20:53:43.461503 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461486 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97"} err="failed to get container status \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": rpc error: code = NotFound desc = could not find container \"2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97\": container with ID starting with 2407cd6678dc880d91fda3089bed2c2ef551b85e59fe28ee09f69d94bd7a5b97 not found: ID does not exist" Oct 30 20:53:43.461571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461503 4988 scope.go:122] "RemoveContainer" containerID="88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc" Oct 30 20:53:43.461760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.461738 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc"} err="failed to get container status \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": rpc error: code = NotFound desc = could not find container \"88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc\": container with ID starting with 88e3765502babfb981203db793c7b040181dd00e69027fbca212b75150b5c8fc not found: ID does not exist" Oct 30 20:53:43.467031 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467009 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca\") pod \"d48c3d99-db81-411b-8701-5974fcbcc399\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " Oct 30 20:53:43.467135 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467072 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca\") pod \"d48c3d99-db81-411b-8701-5974fcbcc399\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " Oct 30 20:53:43.467135 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467107 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config\") pod \"d48c3d99-db81-411b-8701-5974fcbcc399\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " Oct 30 20:53:43.467238 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467133 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs\" (UniqueName: \"kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs\") pod \"d48c3d99-db81-411b-8701-5974fcbcc399\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " Oct 30 20:53:43.467238 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467161 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls\" (UniqueName: \"kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls\") pod \"d48c3d99-db81-411b-8701-5974fcbcc399\" (UID: \"d48c3d99-db81-411b-8701-5974fcbcc399\") " Oct 30 20:53:43.467505 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467403 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca" pod "d48c3d99-db81-411b-8701-5974fcbcc399" (UID: "d48c3d99-db81-411b-8701-5974fcbcc399"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:43.467505 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467493 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config" pod "d48c3d99-db81-411b-8701-5974fcbcc399" (UID: "d48c3d99-db81-411b-8701-5974fcbcc399"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:43.467793 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.467772 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca" pod "d48c3d99-db81-411b-8701-5974fcbcc399" (UID: "d48c3d99-db81-411b-8701-5974fcbcc399"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:43.469187 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.469166 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs" pod "d48c3d99-db81-411b-8701-5974fcbcc399" (UID: "d48c3d99-db81-411b-8701-5974fcbcc399"). InnerVolumeSpecName "kube-api-access-hkkqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:43.469255 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.469219 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls" pod "d48c3d99-db81-411b-8701-5974fcbcc399" (UID: "d48c3d99-db81-411b-8701-5974fcbcc399"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:43.568061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568030 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls\" (UniqueName: \"kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls\") pod \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " Oct 30 20:53:43.568257 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568122 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config\" (UniqueName: \"kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config\") pod \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " Oct 30 20:53:43.568257 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568148 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f\" (UniqueName: \"kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f\") pod \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\" (UID: \"db1a7232-073a-47c4-8b5e-a4ddab7f7589\") " Oct 30 20:53:43.568375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568298 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.568375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568308 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.568375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568317 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d48c3d99-db81-411b-8701-5974fcbcc399-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.568375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568326 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkkqs\" (UniqueName: \"kubernetes.io/projected/d48c3d99-db81-411b-8701-5974fcbcc399-kube-api-access-hkkqs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.568375 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568336 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d48c3d99-db81-411b-8701-5974fcbcc399-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.568758 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.568465 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config" pod "db1a7232-073a-47c4-8b5e-a4ddab7f7589" (UID: "db1a7232-073a-47c4-8b5e-a4ddab7f7589"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:43.570149 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.570126 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f" pod "db1a7232-073a-47c4-8b5e-a4ddab7f7589" (UID: "db1a7232-073a-47c4-8b5e-a4ddab7f7589"). InnerVolumeSpecName "kube-api-access-qsn5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:43.570149 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.570133 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls" pod "db1a7232-073a-47c4-8b5e-a4ddab7f7589" (UID: "db1a7232-073a-47c4-8b5e-a4ddab7f7589"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:43.669267 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.669229 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsn5f\" (UniqueName: \"kubernetes.io/projected/db1a7232-073a-47c4-8b5e-a4ddab7f7589-kube-api-access-qsn5f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.669267 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.669256 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/db1a7232-073a-47c4-8b5e-a4ddab7f7589-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.669267 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.669266 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db1a7232-073a-47c4-8b5e-a4ddab7f7589-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:43.854190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.854154 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5"] Oct 30 20:53:43.861195 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:43.861163 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x76h5"] Oct 30 20:53:44.015745 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.015660 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" path="/var/lib/kubelet/pods/d48c3d99-db81-411b-8701-5974fcbcc399/volumes" Oct 30 20:53:44.121474 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.121440 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:44.318497 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.318375 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:44.400360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.400327 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 20:53:44.400731 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.400630 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="prometheus" containerID="cri-o://8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" gracePeriod=600 Oct 30 20:53:44.400731 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.400698 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="thanos-sidecar" containerID="cri-o://161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" gracePeriod=600 Oct 30 20:53:44.400938 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.400807 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="config-reloader" containerID="cri-o://6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" gracePeriod=600 Oct 30 20:53:44.418501 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.418473 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:53:44.418789 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.418763 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="alertmanager" containerID="cri-o://6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df" gracePeriod=120 Oct 30 20:53:44.418858 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.418834 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="config-reloader" containerID="cri-o://9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6" gracePeriod=120 Oct 30 20:53:44.439074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.439039 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" event={"ID":"db1a7232-073a-47c4-8b5e-a4ddab7f7589","Type":"ContainerDied","Data":"a7303c9402b950612265ae64683106ba318490bf6146e24750672c90a74e445f"} Oct 30 20:53:44.439209 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.439126 4988 scope.go:122] "RemoveContainer" containerID="49a5972451ab2e5682a6cb2cd3c7d03466181d75da6fd057637976264e409c5e" Oct 30 20:53:44.439209 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.439047 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr" Oct 30 20:53:44.475163 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.475122 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:44.531988 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.531964 4988 scope.go:122] "RemoveContainer" containerID="33afa49a3583a04aada279a2fc06e10f4f3d1343553f26cccb5481f7c0174281" Oct 30 20:53:44.539766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.539748 4988 scope.go:122] "RemoveContainer" containerID="0cb3846c5e64291b4cd5596f0f0dd908956e047c924db94a31e17a059134d7a0" Oct 30 20:53:44.641543 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.641510 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr"] Oct 30 20:53:44.647019 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.646986 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-dvxcr"] Oct 30 20:53:44.765721 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.765688 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:44.917852 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:44.917788 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:45.263776 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.263748 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:45.385402 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385378 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385424 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385468 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385488 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385528 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385560 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385586 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385648 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385739 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385682 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385741 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:45.385937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385776 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385810 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385864 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.385937 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.385897 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca\") pod \"251d4f40-968e-4e98-827e-0ccc962d07b4\" (UID: \"251d4f40-968e-4e98-827e-0ccc962d07b4\") " Oct 30 20:53:45.386186 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.386033 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:45.386237 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.386224 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.386290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.386245 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.386564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.386538 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:45.388192 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.388161 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.388333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.388311 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "kube-api-access-4cvvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:45.388547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.388520 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.388638 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.388537 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.388697 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.388677 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:45.389116 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.389089 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:45.390344 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.390317 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.390467 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.390405 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.391321 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.391300 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:53:45.398560 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.398536 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:53:45.402483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.402461 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config" pod "251d4f40-968e-4e98-827e-0ccc962d07b4" (UID: "251d4f40-968e-4e98-827e-0ccc962d07b4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.447631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.447591 4988 generic.go:356] "Generic (PLEG): container finished" podID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerID="161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" exitCode=0 Oct 30 20:53:45.447631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.447623 4988 generic.go:356] "Generic (PLEG): container finished" podID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerID="6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" exitCode=0 Oct 30 20:53:45.447631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.447633 4988 generic.go:356] "Generic (PLEG): container finished" podID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerID="8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" exitCode=0 Oct 30 20:53:45.447870 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.447857 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 20:53:45.448635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.448607 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerDied","Data":"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602"} Oct 30 20:53:45.448743 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.448715 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerDied","Data":"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900"} Oct 30 20:53:45.448799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.448755 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerDied","Data":"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446"} Oct 30 20:53:45.448799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.448794 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"251d4f40-968e-4e98-827e-0ccc962d07b4","Type":"ContainerDied","Data":"cb122b2099d411b6ff858be9744c3d6fefaed79d38040d0e98c0bf4e0c2c98d2"} Oct 30 20:53:45.448897 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.448837 4988 scope.go:122] "RemoveContainer" containerID="161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" Oct 30 20:53:45.453544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.453390 4988 generic.go:356] "Generic (PLEG): container finished" podID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerID="9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6" exitCode=0 Oct 30 20:53:45.453544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.453432 4988 generic.go:356] "Generic (PLEG): container finished" podID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerID="6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df" exitCode=0 Oct 30 20:53:45.453544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.453451 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerDied","Data":"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6"} Oct 30 20:53:45.453544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.453470 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:53:45.453544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.453507 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerDied","Data":"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df"} Oct 30 20:53:45.453544 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.453533 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ef4d83ee-d9b1-4059-9fdc-dbb627a62178","Type":"ContainerDied","Data":"48132e325d476942124153cf4ff166483bd99b3d3ee20b84d21b9d72cea1a25d"} Oct 30 20:53:45.458716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.458597 4988 scope.go:122] "RemoveContainer" containerID="6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" Oct 30 20:53:45.469034 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.469014 4988 scope.go:122] "RemoveContainer" containerID="8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" Oct 30 20:53:45.476745 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.476722 4988 scope.go:122] "RemoveContainer" containerID="3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27" Oct 30 20:53:45.484750 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.484732 4988 scope.go:122] "RemoveContainer" containerID="161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" Oct 30 20:53:45.485015 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.484999 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": container with ID starting with 161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602 not found: ID does not exist" containerID="161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" Oct 30 20:53:45.485064 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485024 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602"} err="failed to get container status \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": rpc error: code = NotFound desc = could not find container \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": container with ID starting with 161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602 not found: ID does not exist" Oct 30 20:53:45.485064 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485041 4988 scope.go:122] "RemoveContainer" containerID="6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" Oct 30 20:53:45.485277 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.485254 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": container with ID starting with 6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900 not found: ID does not exist" containerID="6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" Oct 30 20:53:45.485345 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485287 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900"} err="failed to get container status \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": rpc error: code = NotFound desc = could not find container \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": container with ID starting with 6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900 not found: ID does not exist" Oct 30 20:53:45.485345 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485310 4988 scope.go:122] "RemoveContainer" containerID="8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" Oct 30 20:53:45.485597 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.485582 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": container with ID starting with 8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446 not found: ID does not exist" containerID="8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" Oct 30 20:53:45.485640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485600 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446"} err="failed to get container status \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": rpc error: code = NotFound desc = could not find container \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": container with ID starting with 8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446 not found: ID does not exist" Oct 30 20:53:45.485640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485614 4988 scope.go:122] "RemoveContainer" containerID="3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27" Oct 30 20:53:45.485823 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.485805 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": container with ID starting with 3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27 not found: ID does not exist" containerID="3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27" Oct 30 20:53:45.485887 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485830 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27"} err="failed to get container status \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": rpc error: code = NotFound desc = could not find container \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": container with ID starting with 3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27 not found: ID does not exist" Oct 30 20:53:45.485887 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.485849 4988 scope.go:122] "RemoveContainer" containerID="161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" Oct 30 20:53:45.486072 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486054 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602"} err="failed to get container status \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": rpc error: code = NotFound desc = could not find container \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": container with ID starting with 161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602 not found: ID does not exist" Oct 30 20:53:45.486121 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486073 4988 scope.go:122] "RemoveContainer" containerID="6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" Oct 30 20:53:45.486301 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486284 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900"} err="failed to get container status \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": rpc error: code = NotFound desc = could not find container \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": container with ID starting with 6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900 not found: ID does not exist" Oct 30 20:53:45.486369 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486302 4988 scope.go:122] "RemoveContainer" containerID="8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" Oct 30 20:53:45.486464 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486447 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486540 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486525 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486544 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446"} err="failed to get container status \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": rpc error: code = NotFound desc = could not find container \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": container with ID starting with 8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446 not found: ID does not exist" Oct 30 20:53:45.486603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486567 4988 scope.go:122] "RemoveContainer" containerID="3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27" Oct 30 20:53:45.486603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486554 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486644 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486676 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486863 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486754 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486863 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486807 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8\") pod \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\" (UID: \"ef4d83ee-d9b1-4059-9fdc-dbb627a62178\") " Oct 30 20:53:45.486971 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486929 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27"} err="failed to get container status \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": rpc error: code = NotFound desc = could not find container \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": container with ID starting with 3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27 not found: ID does not exist" Oct 30 20:53:45.486971 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.486956 4988 scope.go:122] "RemoveContainer" containerID="161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487144 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487151 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/251d4f40-968e-4e98-827e-0ccc962d07b4-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487199 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487221 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487238 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487253 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487269 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487285 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvvm\" (UniqueName: \"kubernetes.io/projected/251d4f40-968e-4e98-827e-0ccc962d07b4-kube-api-access-4cvvm\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487322 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487338 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/251d4f40-968e-4e98-827e-0ccc962d07b4-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487354 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487970 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487371 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/251d4f40-968e-4e98-827e-0ccc962d07b4-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.487970 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487759 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602"} err="failed to get container status \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": rpc error: code = NotFound desc = could not find container \"161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602\": container with ID starting with 161c84d6e75b20eb78edd4943cc0c07521b4b3350ca9a1fa03748d4af6043602 not found: ID does not exist" Oct 30 20:53:45.487970 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.487783 4988 scope.go:122] "RemoveContainer" containerID="6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900" Oct 30 20:53:45.488131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488062 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900"} err="failed to get container status \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": rpc error: code = NotFound desc = could not find container \"6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900\": container with ID starting with 6afbfc7239ab4c67b0042d8cee10a916af9f20ef40ad54dd3884e367299e3900 not found: ID does not exist" Oct 30 20:53:45.488131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488085 4988 scope.go:122] "RemoveContainer" containerID="8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446" Oct 30 20:53:45.488478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488391 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446"} err="failed to get container status \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": rpc error: code = NotFound desc = could not find container \"8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446\": container with ID starting with 8f0a2847f3b510df1b9ab7d28c344ce2b56f05bd7afb4c68c0058eba68fe4446 not found: ID does not exist" Oct 30 20:53:45.488478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488444 4988 scope.go:122] "RemoveContainer" containerID="3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27" Oct 30 20:53:45.488763 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488737 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27"} err="failed to get container status \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": rpc error: code = NotFound desc = could not find container \"3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27\": container with ID starting with 3c9c93443cd1e4f86d0d54ef0c2e84c632f98e05f39cf381f141a43b51e2fe27 not found: ID does not exist" Oct 30 20:53:45.488857 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488764 4988 scope.go:122] "RemoveContainer" containerID="9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6" Oct 30 20:53:45.488946 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.488926 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:45.489512 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.489482 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:53:45.490139 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.490103 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.490228 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.490161 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "kube-api-access-6wrg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:45.494172 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.494127 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.498207 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.498183 4988 scope.go:122] "RemoveContainer" containerID="6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df" Oct 30 20:53:45.499967 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.499948 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:53:45.502645 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.502604 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config" pod "ef4d83ee-d9b1-4059-9fdc-dbb627a62178" (UID: "ef4d83ee-d9b1-4059-9fdc-dbb627a62178"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:45.507695 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.507568 4988 scope.go:122] "RemoveContainer" containerID="2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6" Oct 30 20:53:45.515773 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.515751 4988 scope.go:122] "RemoveContainer" containerID="9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6" Oct 30 20:53:45.516046 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.516026 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6\": container with ID starting with 9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6 not found: ID does not exist" containerID="9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6" Oct 30 20:53:45.516097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516057 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6"} err="failed to get container status \"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6\": rpc error: code = NotFound desc = could not find container \"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6\": container with ID starting with 9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6 not found: ID does not exist" Oct 30 20:53:45.516097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516077 4988 scope.go:122] "RemoveContainer" containerID="6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df" Oct 30 20:53:45.516340 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.516320 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df\": container with ID starting with 6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df not found: ID does not exist" containerID="6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df" Oct 30 20:53:45.516404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516349 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df"} err="failed to get container status \"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df\": rpc error: code = NotFound desc = could not find container \"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df\": container with ID starting with 6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df not found: ID does not exist" Oct 30 20:53:45.516404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516371 4988 scope.go:122] "RemoveContainer" containerID="2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6" Oct 30 20:53:45.516617 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:45.516602 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6\": container with ID starting with 2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6 not found: ID does not exist" containerID="2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6" Oct 30 20:53:45.516665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516620 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6"} err="failed to get container status \"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6\": rpc error: code = NotFound desc = could not find container \"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6\": container with ID starting with 2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6 not found: ID does not exist" Oct 30 20:53:45.516665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516631 4988 scope.go:122] "RemoveContainer" containerID="9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6" Oct 30 20:53:45.516844 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516826 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6"} err="failed to get container status \"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6\": rpc error: code = NotFound desc = could not find container \"9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6\": container with ID starting with 9b549c4879dfa73fed832dc1107a4e580093a210606a0e1f6820ad9f2f278ee6 not found: ID does not exist" Oct 30 20:53:45.516911 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.516845 4988 scope.go:122] "RemoveContainer" containerID="6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df" Oct 30 20:53:45.517080 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.517062 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df"} err="failed to get container status \"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df\": rpc error: code = NotFound desc = could not find container \"6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df\": container with ID starting with 6b651be18dbe39d00efff026f6be430b1953f301a2997bf7457a14b45c3120df not found: ID does not exist" Oct 30 20:53:45.517125 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.517081 4988 scope.go:122] "RemoveContainer" containerID="2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6" Oct 30 20:53:45.517266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.517251 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6"} err="failed to get container status \"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6\": rpc error: code = NotFound desc = could not find container \"2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6\": container with ID starting with 2ba2b529d7dcb81fed11d8155eedc897caacb75aae5dab66ea5fec45534d1ed6 not found: ID does not exist" Oct 30 20:53:45.588308 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588264 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.588308 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588299 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.588308 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588312 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.588582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588324 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.588582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588338 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 20:53:45.588582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588352 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.588582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588364 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrg8\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-kube-api-access-6wrg8\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.588582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.588377 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef4d83ee-d9b1-4059-9fdc-dbb627a62178-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.722621 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.722585 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 20:53:45.733598 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.733564 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 20:53:45.877367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.877331 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:53:45.885771 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:45.885743 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:53:46.015693 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:46.015614 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" path="/var/lib/kubelet/pods/251d4f40-968e-4e98-827e-0ccc962d07b4/volumes" Oct 30 20:53:46.016129 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:46.016116 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" path="/var/lib/kubelet/pods/db1a7232-073a-47c4-8b5e-a4ddab7f7589/volumes" Oct 30 20:53:46.016738 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:46.016719 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" path="/var/lib/kubelet/pods/ef4d83ee-d9b1-4059-9fdc-dbb627a62178/volumes" Oct 30 20:53:46.932470 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:46.932437 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:53:46.932946 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:46.932636 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="fa82a77c-2bab-49ee-a17b-49b56f8ff243" containerName="otc-container" containerID="cri-o://1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e" gracePeriod=30 Oct 30 20:53:46.947766 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:46.947731 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:47.093665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.093615 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:47.182348 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.182320 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:53:47.303909 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.303826 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs\") pod \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " Oct 30 20:53:47.303909 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.303889 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs\") pod \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " Oct 30 20:53:47.304107 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.304034 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8\" (UniqueName: \"kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8\") pod \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " Oct 30 20:53:47.304164 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.304104 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal\") pod \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " Oct 30 20:53:47.304164 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.304158 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs\") pod \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " Oct 30 20:53:47.304267 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.304189 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert\") pod \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\" (UID: \"fa82a77c-2bab-49ee-a17b-49b56f8ff243\") " Oct 30 20:53:47.304554 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.304524 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal" pod "fa82a77c-2bab-49ee-a17b-49b56f8ff243" (UID: "fa82a77c-2bab-49ee-a17b-49b56f8ff243"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:47.306117 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.306088 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8" pod "fa82a77c-2bab-49ee-a17b-49b56f8ff243" (UID: "fa82a77c-2bab-49ee-a17b-49b56f8ff243"). InnerVolumeSpecName "kube-api-access-d2rj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:47.306246 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.306216 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs" pod "fa82a77c-2bab-49ee-a17b-49b56f8ff243" (UID: "fa82a77c-2bab-49ee-a17b-49b56f8ff243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:47.306377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.306244 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs" pod "fa82a77c-2bab-49ee-a17b-49b56f8ff243" (UID: "fa82a77c-2bab-49ee-a17b-49b56f8ff243"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:47.306377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.306275 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert" pod "fa82a77c-2bab-49ee-a17b-49b56f8ff243" (UID: "fa82a77c-2bab-49ee-a17b-49b56f8ff243"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:47.307103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.307084 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs" pod "fa82a77c-2bab-49ee-a17b-49b56f8ff243" (UID: "fa82a77c-2bab-49ee-a17b-49b56f8ff243"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:47.371593 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.371559 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:47.405201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.405170 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.405201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.405196 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.405201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.405206 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.405439 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.405216 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.405439 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.405225 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa82a77c-2bab-49ee-a17b-49b56f8ff243-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.405439 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.405233 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2rj8\" (UniqueName: \"kubernetes.io/projected/fa82a77c-2bab-49ee-a17b-49b56f8ff243-kube-api-access-d2rj8\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:47.466268 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.466234 4988 generic.go:356] "Generic (PLEG): container finished" podID="fa82a77c-2bab-49ee-a17b-49b56f8ff243" containerID="1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e" exitCode=0 Oct 30 20:53:47.466462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.466275 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:53:47.466462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.466278 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fa82a77c-2bab-49ee-a17b-49b56f8ff243","Type":"ContainerDied","Data":"1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e"} Oct 30 20:53:47.466462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.466377 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fa82a77c-2bab-49ee-a17b-49b56f8ff243","Type":"ContainerDied","Data":"2f8a1af7701fdf2c57c8d457c13e0bc142db6cec7b5b7e5254add37619938948"} Oct 30 20:53:47.466462 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.466423 4988 scope.go:122] "RemoveContainer" containerID="1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e" Oct 30 20:53:47.474740 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.474725 4988 scope.go:122] "RemoveContainer" containerID="1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e" Oct 30 20:53:47.475005 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:47.474986 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e\": container with ID starting with 1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e not found: ID does not exist" containerID="1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e" Oct 30 20:53:47.475076 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.475011 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e"} err="failed to get container status \"1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e\": rpc error: code = NotFound desc = could not find container \"1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e\": container with ID starting with 1ff2ba5de100d62734170518b0257218ff27ccf0e42ba34570b93b731babe16e not found: ID does not exist" Oct 30 20:53:47.630945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.630869 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:53:47.635762 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:47.635738 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:53:48.016705 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.016675 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa82a77c-2bab-49ee-a17b-49b56f8ff243" path="/var/lib/kubelet/pods/fa82a77c-2bab-49ee-a17b-49b56f8ff243/volumes" Oct 30 20:53:48.048483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.048450 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:53:48.048703 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.048673 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo" containerID="cri-o://acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe" gracePeriod=30 Oct 30 20:53:48.048867 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.048701 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway" containerID="cri-o://84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20" gracePeriod=30 Oct 30 20:53:48.048867 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.048711 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway-opa" containerID="cri-o://5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd" gracePeriod=30 Oct 30 20:53:48.098268 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.098229 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:48.196970 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.196920 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.48:8083/ready\": dial tcp 10.132.0.48:8083: connect: connection refused" Oct 30 20:53:48.200663 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.200633 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:53:48.473163 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.473129 4988 generic.go:356] "Generic (PLEG): container finished" podID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerID="5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd" exitCode=0 Oct 30 20:53:48.473163 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.473158 4988 generic.go:356] "Generic (PLEG): container finished" podID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerID="acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe" exitCode=0 Oct 30 20:53:48.473360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.473200 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerDied","Data":"5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd"} Oct 30 20:53:48.473360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.473252 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerDied","Data":"acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe"} Oct 30 20:53:48.779998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.779967 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:48.940120 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:48.940084 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:49.103360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.103230 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:49.261985 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.261954 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:49.296650 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.296616 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:53:49.424157 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424121 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424175 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424280 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424325 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424380 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424437 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424481 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424668 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424532 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5\" (UniqueName: \"kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424720 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424573 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:49.424720 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424585 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls\") pod \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\" (UID: \"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57\") " Oct 30 20:53:49.424994 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.424973 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.426889 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.425173 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:49.427037 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.427016 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:49.427533 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.427490 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:49.427533 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.427509 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:49.427680 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.427603 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:49.427985 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.427938 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:49.430448 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.430398 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:53:49.435735 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.435710 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5" pod "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" (UID: "6af6e281-bed5-48e3-a1c3-c0d75c4d5e57"). InnerVolumeSpecName "kube-api-access-cf8x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:49.480313 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.480274 4988 generic.go:356] "Generic (PLEG): container finished" podID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerID="84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20" exitCode=0 Oct 30 20:53:49.480482 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.480342 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:53:49.480482 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.480365 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerDied","Data":"84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20"} Oct 30 20:53:49.480482 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.480467 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6af6e281-bed5-48e3-a1c3-c0d75c4d5e57","Type":"ContainerDied","Data":"9e4cca948fc4d138fe6e3b631ced40a4743e851dded0ede996a6d3bb7908b957"} Oct 30 20:53:49.480608 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.480492 4988 scope.go:122] "RemoveContainer" containerID="5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd" Oct 30 20:53:49.489372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.489352 4988 scope.go:122] "RemoveContainer" containerID="84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20" Oct 30 20:53:49.496865 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.496847 4988 scope.go:122] "RemoveContainer" containerID="acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe" Oct 30 20:53:49.504025 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.504003 4988 scope.go:122] "RemoveContainer" containerID="5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd" Oct 30 20:53:49.504294 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:49.504274 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd\": container with ID starting with 5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd not found: ID does not exist" containerID="5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd" Oct 30 20:53:49.504354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.504302 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd"} err="failed to get container status \"5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd\": rpc error: code = NotFound desc = could not find container \"5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd\": container with ID starting with 5728c149893d814c54b51a9898c9caa72e52634c428e2760787906832c137fcd not found: ID does not exist" Oct 30 20:53:49.504354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.504321 4988 scope.go:122] "RemoveContainer" containerID="84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20" Oct 30 20:53:49.504597 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:49.504573 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20\": container with ID starting with 84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20 not found: ID does not exist" containerID="84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20" Oct 30 20:53:49.504639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.504603 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20"} err="failed to get container status \"84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20\": rpc error: code = NotFound desc = could not find container \"84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20\": container with ID starting with 84ed9ad22bfa78d1b025d16628fcc1990bfdd82430ce18ba765ed1d611970c20 not found: ID does not exist" Oct 30 20:53:49.504639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.504617 4988 scope.go:122] "RemoveContainer" containerID="acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe" Oct 30 20:53:49.504863 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:49.504846 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe\": container with ID starting with acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe not found: ID does not exist" containerID="acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe" Oct 30 20:53:49.504910 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.504868 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe"} err="failed to get container status \"acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe\": rpc error: code = NotFound desc = could not find container \"acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe\": container with ID starting with acfed5ad166cc5b71cfcb8a1efee22f94af470494a506f776e7a268f06c92cfe not found: ID does not exist" Oct 30 20:53:49.526303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526276 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526303 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526304 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526460 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526315 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526460 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526326 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526460 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526341 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf8x5\" (UniqueName: \"kubernetes.io/projected/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-kube-api-access-cf8x5\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526460 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526354 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526460 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526363 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:53:49.526460 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.526392 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:53:49.534969 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.534953 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:53:49.627189 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.627154 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 20:53:49.655325 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.655289 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:53:49.660209 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:49.660182 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:53:50.015965 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:53:50.015932 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" path="/var/lib/kubelet/pods/6af6e281-bed5-48e3-a1c3-c0d75c4d5e57/volumes" Oct 30 20:53:56.392947 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:53:56.392906 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:54:03.005273 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:03.005234 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:03.185273 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:03.185239 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:03.362087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:03.361994 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:03.575047 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:03.575013 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:03.755132 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:03.755081 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:04.068564 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068487 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5"] Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068802 4988 cpu_manager.go:414] "containerMap: removing container" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="kube-rbac-proxy" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068812 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="kube-rbac-proxy" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068823 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="init-config-reloader" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068827 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="init-config-reloader" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068841 4988 cpu_manager.go:414] "containerMap: removing container" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="prom-label-proxy" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068846 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="prom-label-proxy" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068852 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068856 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068864 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068869 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068878 4988 cpu_manager.go:414] "containerMap: removing container" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="thanos-sidecar" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068882 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="thanos-sidecar" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068889 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="config-reloader" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068893 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="config-reloader" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068901 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="method-gate" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068905 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="method-gate" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068911 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="alertmanager" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068915 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="alertmanager" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068922 4988 cpu_manager.go:414] "containerMap: removing container" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="init-config-reloader" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068926 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="init-config-reloader" Oct 30 20:54:04.068923 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068933 4988 cpu_manager.go:414] "containerMap: removing container" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="prometheus" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068937 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="prometheus" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068953 4988 cpu_manager.go:414] "containerMap: removing container" podUID="fa82a77c-2bab-49ee-a17b-49b56f8ff243" containerName="otc-container" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068957 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa82a77c-2bab-49ee-a17b-49b56f8ff243" containerName="otc-container" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068964 4988 cpu_manager.go:414] "containerMap: removing container" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="config-reloader" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068969 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="config-reloader" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068975 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="kube-rbac-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068979 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="kube-rbac-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068986 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="prom-label-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068990 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="prom-label-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068995 4988 cpu_manager.go:414] "containerMap: removing container" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="method-gate" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.068999 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="method-gate" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069004 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway-opa" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069008 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway-opa" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069085 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa82a77c-2bab-49ee-a17b-49b56f8ff243" containerName="otc-container" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069092 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="config-reloader" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069097 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="kube-rbac-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069102 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="prom-label-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069106 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="method-gate" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069112 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway-opa" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069118 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="kube-rbac-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069123 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1a7232-073a-47c4-8b5e-a4ddab7f7589" containerName="prom-label-proxy" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069128 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069133 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af6e281-bed5-48e3-a1c3-c0d75c4d5e57" containerName="tempo-gateway" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069138 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="thanos-sidecar" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069143 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="config-reloader" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069148 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48c3d99-db81-411b-8701-5974fcbcc399" containerName="method-gate" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069152 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4d83ee-d9b1-4059-9fdc-dbb627a62178" containerName="alertmanager" Oct 30 20:54:04.069521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.069158 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d4f40-968e-4e98-827e-0ccc962d07b4" containerName="prometheus" Oct 30 20:54:04.072371 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.072355 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.095650 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.095617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:04.125563 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.125527 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:54:04.139711 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.139680 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxf6\" (UniqueName: \"kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.139900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.139828 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.139900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.139867 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.139900 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.139890 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.140035 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.139957 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.174262 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.174220 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 20:54:04.180485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.174654 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 20:54:04.180485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.174781 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-w8jn8\"" Oct 30 20:54:04.180485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.174838 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 20:54:04.180485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.174828 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 20:54:04.180485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.175364 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:54:04.180485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.178136 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5"] Oct 30 20:54:04.185803 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.185776 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh"] Oct 30 20:54:04.189297 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.189281 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.240887 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.240847 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.240887 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.240882 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.241115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.240900 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.241115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.240924 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.241115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.240999 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhfn\" (UniqueName: \"kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.241115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.241045 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxf6\" (UniqueName: \"kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.241115 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.241088 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.241377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.241124 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.241593 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.241565 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.241593 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.241588 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.241753 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.241726 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.243500 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.243477 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.244443 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.244427 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 20:54:04.244992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.244979 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 20:54:04.245058 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.244989 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-krkkg\"" Oct 30 20:54:04.265531 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.265496 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh"] Oct 30 20:54:04.311379 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.311346 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxf6\" (UniqueName: \"kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.325575 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.325477 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:04.342001 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.341968 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhfn\" (UniqueName: \"kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.342166 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.342026 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.342166 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.342068 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.342850 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.342828 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.344558 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.344540 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.389726 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.389688 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 20:54:04.404127 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.404098 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhfn\" (UniqueName: \"kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn\") pod \"data-science-thanos-querier-proxy-77657866bd-xfhhh\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.506914 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.506874 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 20:54:04.776825 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.776791 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5"] Oct 30 20:54:04.780894 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:54:04.780865 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d80f20_1d3c_44bd_99c9_3d2ed1949f8d.slice/crio-d94abf5ead9115bf2468b112a8a422ebe60df8ef95aa14453ba02c4b79e996f7 WatchSource:0}: Error finding container d94abf5ead9115bf2468b112a8a422ebe60df8ef95aa14453ba02c4b79e996f7: Status 404 returned error can't find the container with id d94abf5ead9115bf2468b112a8a422ebe60df8ef95aa14453ba02c4b79e996f7 Oct 30 20:54:04.881733 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:04.881702 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh"] Oct 30 20:54:04.884479 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:54:04.884450 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd1d521_f511_4f19_993d_38c4e069d692.slice/crio-2fa3de3c1fb0307fe5c96419d17416ebef6911c18390476f89ca2462f30d51d3 WatchSource:0}: Error finding container 2fa3de3c1fb0307fe5c96419d17416ebef6911c18390476f89ca2462f30d51d3: Status 404 returned error can't find the container with id 2fa3de3c1fb0307fe5c96419d17416ebef6911c18390476f89ca2462f30d51d3 Oct 30 20:54:05.553950 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.553909 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerStarted","Data":"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661"} Oct 30 20:54:05.554392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.553984 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerStarted","Data":"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c"} Oct 30 20:54:05.554392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.554029 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerStarted","Data":"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0"} Oct 30 20:54:05.554392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.554067 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerStarted","Data":"2fa3de3c1fb0307fe5c96419d17416ebef6911c18390476f89ca2462f30d51d3"} Oct 30 20:54:05.556033 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.556007 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerStarted","Data":"7c528163431571df9a186d44b41e3c57d010105581dc8595069654b9d6b7ca67"} Oct 30 20:54:05.556139 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.556059 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerStarted","Data":"739875c010f91b5f7d557c0bae7a5559893316a727be60fb0889c88c275a7dea"} Oct 30 20:54:05.556139 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.556084 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerStarted","Data":"fc7e7f15d66a013016a7e7eb64b108c28cb8d86d9bcf206359a9e4a6f02b30e9"} Oct 30 20:54:05.556139 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.556106 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerStarted","Data":"d94abf5ead9115bf2468b112a8a422ebe60df8ef95aa14453ba02c4b79e996f7"} Oct 30 20:54:05.685527 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.685470 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" podStartSLOduration=1.6854547069999999 podStartE2EDuration="1.685454707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:05.684202014 +0000 UTC m=+1073.791697924" watchObservedRunningTime="2026-10-30 20:54:05.685454707 +0000 UTC m=+1073.792950611" Oct 30 20:54:05.779087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.779051 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:05.811841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:05.811733 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" podStartSLOduration=1.811719547 podStartE2EDuration="1.811719547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:05.810844656 +0000 UTC m=+1073.918340568" watchObservedRunningTime="2026-10-30 20:54:05.811719547 +0000 UTC m=+1073.919215451" Oct 30 20:54:07.465335 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:07.465299 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:08.947469 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.947433 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:54:08.947903 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.947617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:08.950925 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.950910 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:08.981194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.981163 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:08.981354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.981200 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:08.981354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.981219 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:08.981354 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.981289 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:08.981478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.981385 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:08.981478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:08.981431 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.013859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.013824 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:54:09.054842 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.054803 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:54:09.055014 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.054844 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:54:09.055014 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.054882 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:54:09.061426 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.061381 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:54:09.061593 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.061382 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:54:09.061856 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.061840 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 20:54:09.082565 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.082524 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.082565 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.082567 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.082802 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.082589 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.082802 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.082621 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.082802 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.082642 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.082802 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.082664 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.083340 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.083309 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.085248 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.085218 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.085356 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.085278 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.085356 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.085283 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.085516 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.085495 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.152373 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.152339 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c\") pod \"data-science-collector-collector-1\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.167342 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.167303 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:09.263845 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.263761 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:54:09.365131 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.365098 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:09.646756 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.646726 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:54:09.650104 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:54:09.650074 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6effc12c_9e7f_4f9c_935b_5d75a45e6c59.slice/crio-326259cdb971e1f9aee48068b8c202eb4e5901b66a933a9076a17426d3a08b1a WatchSource:0}: Error finding container 326259cdb971e1f9aee48068b8c202eb4e5901b66a933a9076a17426d3a08b1a: Status 404 returned error can't find the container with id 326259cdb971e1f9aee48068b8c202eb4e5901b66a933a9076a17426d3a08b1a Oct 30 20:54:09.652275 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.652245 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:54:09.652334 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:09.652308 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:54:10.582140 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:10.582099 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6effc12c-9e7f-4f9c-935b-5d75a45e6c59","Type":"ContainerStarted","Data":"b8ce6b4f4bd9ac1be557ee2c6b699dd31cbb37527b678237c97a59c8f0e9ae55"} Oct 30 20:54:10.582588 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:10.582161 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6effc12c-9e7f-4f9c-935b-5d75a45e6c59","Type":"ContainerStarted","Data":"326259cdb971e1f9aee48068b8c202eb4e5901b66a933a9076a17426d3a08b1a"} Oct 30 20:54:10.701049 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:10.700990 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.700977409 podStartE2EDuration="2.700977409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:10.699544602 +0000 UTC m=+1078.807040506" watchObservedRunningTime="2026-10-30 20:54:10.700977409 +0000 UTC m=+1078.808473319" Oct 30 20:54:21.327862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:21.327822 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:21.769281 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:21.769237 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:21.920382 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:21.920342 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:22.082713 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.082618 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:22.575097 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.575062 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:22.706741 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.706706 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:54:22.712326 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.712300 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.732082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.732050 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:22.767712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.767667 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 20:54:22.768000 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.767973 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 20:54:22.768366 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.768347 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 20:54:22.817353 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817319 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 20:54:22.817580 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817505 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 20:54:22.817580 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817516 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 20:54:22.817580 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817566 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 20:54:22.817709 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817603 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-x6vfm\"" Oct 30 20:54:22.817709 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817667 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 20:54:22.817709 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.817694 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 20:54:22.825053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.825026 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:54:22.826233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.826177 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:54:22.829912 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.829889 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.885547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.885511 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:54:22.897029 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.896993 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897034 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897063 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mw92\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897202 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897316 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897377 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897349 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897519 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897385 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897519 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897473 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897519 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897495 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897519 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897511 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897643 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897547 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.897643 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.897599 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.975030 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.974982 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 20:54:22.975030 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.974983 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 20:54:22.975251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.975052 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-ch6xk\"" Oct 30 20:54:22.975251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.975070 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 20:54:22.975251 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.975111 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 20:54:22.998793 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998752 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.998886 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998797 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.998886 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998823 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.998886 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998870 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.998990 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998929 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mw92\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.998990 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998968 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.998992 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlj89\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.999065 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999043 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999169 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999104 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999169 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999155 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999272 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999188 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.999272 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999230 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999272 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999256 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.999455 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999290 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999455 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999333 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999584 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999511 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999848 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999809 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.999961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999851 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:22.999961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999916 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:22.999961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:22.999953 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.000136 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.000083 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.000801 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.000775 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.002074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.002034 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.002446 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.002406 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.002446 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.002437 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.002873 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.002850 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.002964 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.002897 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.003030 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.002979 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.003266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.003246 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.003335 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.003303 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.061481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.061437 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mw92\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:23.100841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.100749 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.100841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.100798 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.100841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.100830 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.101107 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.100865 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.101107 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.100895 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.101107 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.100915 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.101107 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.101040 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlj89\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.101473 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.101443 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.103372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.103259 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.103372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.103311 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.103372 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.103322 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.103622 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.103605 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.103686 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.103657 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.162890 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.162854 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlj89\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.446989 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.446946 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:23.883049 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:23.883021 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 20:54:24.642245 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:24.642207 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerStarted","Data":"6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32"} Oct 30 20:54:24.642464 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:24.642273 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerStarted","Data":"dcb74adf334962fda088c1f1bd4cce5263a879d3f85cd9d8cb1735786f076261"} Oct 30 20:54:26.394028 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:54:26.393986 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:54:28.662101 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:28.662067 4988 generic.go:356] "Generic (PLEG): container finished" podID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerID="6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32" exitCode=0 Oct 30 20:54:28.662635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:28.662142 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerDied","Data":"6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32"} Oct 30 20:54:29.564469 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.564399 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:29.668858 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.668822 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:29.669308 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.669286 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerStarted","Data":"047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293"} Oct 30 20:54:29.669369 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.669343 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerStarted","Data":"28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64"} Oct 30 20:54:29.669556 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.669538 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:29.766086 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.766036 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:29.766281 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.766197 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:29.799958 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.799899 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=4.413984234 podStartE2EDuration="7.79988379s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385899556s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:29.797846536 +0000 UTC m=+1097.905342448" watchObservedRunningTime="2026-10-30 20:54:29.79988379 +0000 UTC m=+1097.907379700" Oct 30 20:54:29.818640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.818602 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") DevicePath \"csi-468025a65b88095496ffee260c47731d5a0a7d1245692f39a473b974fd0f0891\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:29.959494 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:29.959456 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fb02f91ab75afdcfd8c2ac3eaeb0ba7303c67490d38fe3164f5fc7856f2f1875/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:30.024314 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:30.024275 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:30.232009 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:30.231921 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:30.765852 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:30.765814 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 20:54:30.774080 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:54:30.774044 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeeab09_eb4d_486c_8d3e_b9beca0b768b.slice/crio-7b3639a7329ec048e167eda4d50095b65235cb0d852c7e3033acd67510466643 WatchSource:0}: Error finding container 7b3639a7329ec048e167eda4d50095b65235cb0d852c7e3033acd67510466643: Status 404 returned error can't find the container with id 7b3639a7329ec048e167eda4d50095b65235cb0d852c7e3033acd67510466643 Oct 30 20:54:31.687289 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:31.687235 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerStarted","Data":"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021"} Oct 30 20:54:31.687526 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:31.687359 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerStarted","Data":"7b3639a7329ec048e167eda4d50095b65235cb0d852c7e3033acd67510466643"} Oct 30 20:54:34.703598 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:34.703562 4988 generic.go:356] "Generic (PLEG): container finished" podID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerID="78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021" exitCode=0 Oct 30 20:54:34.704057 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:34.703635 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerDied","Data":"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021"} Oct 30 20:54:35.674892 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:35.674859 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 20:54:35.711090 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:35.711047 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerStarted","Data":"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74"} Oct 30 20:54:35.711504 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:35.711118 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerStarted","Data":"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1"} Oct 30 20:54:35.711504 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:35.711144 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerStarted","Data":"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250"} Oct 30 20:54:35.936548 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:35.936430 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.651552334 podStartE2EDuration="13.936397499s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284845165s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:35.934482939 +0000 UTC m=+1104.041978850" watchObservedRunningTime="2026-10-30 20:54:35.936397499 +0000 UTC m=+1104.043893410" Oct 30 20:54:40.232866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:40.232828 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:45.233152 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:45.233109 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:45.238534 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:45.238513 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:45.760512 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:54:45.760476 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 20:54:56.394752 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:54:56.394671 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:55:00.784427 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:00.784362 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:00.943619 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:00.943570 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:01.094259 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:01.094173 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:01.245042 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:01.245008 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:01.473222 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:01.473187 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:01.635930 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:01.635889 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:05.253667 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:05.253623 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:05.445983 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:05.445940 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:06.611949 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:06.611908 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:55:06.612475 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:06.612118 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6effc12c-9e7f-4f9c-935b-5d75a45e6c59" containerName="otc-container" containerID="cri-o://b8ce6b4f4bd9ac1be557ee2c6b699dd31cbb37527b678237c97a59c8f0e9ae55" gracePeriod=30 Oct 30 20:55:06.965277 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:06.965243 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:07.125433 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:07.125382 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:07.308228 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:07.308133 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:07.804521 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:07.804484 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:08.861697 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:08.861665 4988 generic.go:356] "Generic (PLEG): container finished" podID="6effc12c-9e7f-4f9c-935b-5d75a45e6c59" containerID="b8ce6b4f4bd9ac1be557ee2c6b699dd31cbb37527b678237c97a59c8f0e9ae55" exitCode=0 Oct 30 20:55:08.862088 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:08.861725 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6effc12c-9e7f-4f9c-935b-5d75a45e6c59","Type":"ContainerDied","Data":"b8ce6b4f4bd9ac1be557ee2c6b699dd31cbb37527b678237c97a59c8f0e9ae55"} Oct 30 20:55:08.962531 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:08.962503 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:09.119819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.119725 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert\") pod \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " Oct 30 20:55:09.119819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.119782 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs\") pod \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " Oct 30 20:55:09.119819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.119814 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal\") pod \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " Oct 30 20:55:09.120111 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.119887 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c\") pod \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " Oct 30 20:55:09.120111 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.119970 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs\") pod \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " Oct 30 20:55:09.120111 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.120073 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs\") pod \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\" (UID: \"6effc12c-9e7f-4f9c-935b-5d75a45e6c59\") " Oct 30 20:55:09.120275 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.120208 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal" pod "6effc12c-9e7f-4f9c-935b-5d75a45e6c59" (UID: "6effc12c-9e7f-4f9c-935b-5d75a45e6c59"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:09.120466 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.120441 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.122215 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.122181 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs" pod "6effc12c-9e7f-4f9c-935b-5d75a45e6c59" (UID: "6effc12c-9e7f-4f9c-935b-5d75a45e6c59"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:09.122215 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.122209 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs" pod "6effc12c-9e7f-4f9c-935b-5d75a45e6c59" (UID: "6effc12c-9e7f-4f9c-935b-5d75a45e6c59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:09.122391 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.122331 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c" pod "6effc12c-9e7f-4f9c-935b-5d75a45e6c59" (UID: "6effc12c-9e7f-4f9c-935b-5d75a45e6c59"). InnerVolumeSpecName "kube-api-access-s7b4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:09.122470 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.122397 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert" pod "6effc12c-9e7f-4f9c-935b-5d75a45e6c59" (UID: "6effc12c-9e7f-4f9c-935b-5d75a45e6c59"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:09.122512 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.122482 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs" pod "6effc12c-9e7f-4f9c-935b-5d75a45e6c59" (UID: "6effc12c-9e7f-4f9c-935b-5d75a45e6c59"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:09.221069 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.221010 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.221069 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.221064 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.221069 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.221075 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.221069 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.221086 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7b4c\" (UniqueName: \"kubernetes.io/projected/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-kube-api-access-s7b4c\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.221357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.221097 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6effc12c-9e7f-4f9c-935b-5d75a45e6c59-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.868261 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.868229 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:09.868732 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.868228 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6effc12c-9e7f-4f9c-935b-5d75a45e6c59","Type":"ContainerDied","Data":"326259cdb971e1f9aee48068b8c202eb4e5901b66a933a9076a17426d3a08b1a"} Oct 30 20:55:09.868732 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:09.868363 4988 scope.go:122] "RemoveContainer" containerID="b8ce6b4f4bd9ac1be557ee2c6b699dd31cbb37527b678237c97a59c8f0e9ae55" Oct 30 20:55:10.032557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.032523 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:55:10.082059 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.082019 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:55:10.114019 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.113980 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:55:10.114459 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.114441 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6effc12c-9e7f-4f9c-935b-5d75a45e6c59" containerName="otc-container" Oct 30 20:55:10.114555 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.114459 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6effc12c-9e7f-4f9c-935b-5d75a45e6c59" containerName="otc-container" Oct 30 20:55:10.114611 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.114561 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6effc12c-9e7f-4f9c-935b-5d75a45e6c59" containerName="otc-container" Oct 30 20:55:10.118984 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.118929 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.171519 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.171482 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:55:10.171874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.171829 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 20:55:10.171874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.171832 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:55:10.171874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.171839 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:55:10.171874 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.171858 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:55:10.172153 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.171950 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:55:10.176978 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.176943 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:55:10.229863 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.229823 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.230061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.229881 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.230061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.229952 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9j8z\" (UniqueName: \"kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.230061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.230020 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.230061 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.230043 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.230230 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.230091 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.331362 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.331321 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.331607 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.331381 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.331607 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.331433 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9j8z\" (UniqueName: \"kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.331607 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.331488 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.331607 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.331515 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.331607 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.331550 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.334574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.333008 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.334947 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.334924 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.335062 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.335038 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.341285 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.341246 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.341468 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.341294 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.392611 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.392575 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9j8z\" (UniqueName: \"kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z\") pod \"data-science-collector-collector-1\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.431429 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.431373 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:55:10.807895 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.807815 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:55:10.811332 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:55:10.811294 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd608d9aa_dd15_4bde_b8d5_c11870fdea24.slice/crio-30cda92835986f314a171292fcf3af62f7f71075cd32e87046dc0c2fb7a26809 WatchSource:0}: Error finding container 30cda92835986f314a171292fcf3af62f7f71075cd32e87046dc0c2fb7a26809: Status 404 returned error can't find the container with id 30cda92835986f314a171292fcf3af62f7f71075cd32e87046dc0c2fb7a26809 Oct 30 20:55:10.813380 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.813346 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:55:10.813496 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.813433 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:55:10.875579 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:10.875544 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d608d9aa-dd15-4bde-b8d5-c11870fdea24","Type":"ContainerStarted","Data":"30cda92835986f314a171292fcf3af62f7f71075cd32e87046dc0c2fb7a26809"} Oct 30 20:55:11.883072 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:11.883032 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d608d9aa-dd15-4bde-b8d5-c11870fdea24","Type":"ContainerStarted","Data":"245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee"} Oct 30 20:55:12.001932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:12.001876 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.001863892 podStartE2EDuration="2.001863892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:12.000776633 +0000 UTC m=+1140.108272548" watchObservedRunningTime="2026-10-30 20:55:12.001863892 +0000 UTC m=+1140.109359877" Oct 30 20:55:12.017998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:12.017957 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6effc12c-9e7f-4f9c-935b-5d75a45e6c59" path="/var/lib/kubelet/pods/6effc12c-9e7f-4f9c-935b-5d75a45e6c59/volumes" Oct 30 20:55:19.735631 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:55:19.735589 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:26.394444 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:55:26.394391 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:55:56.393850 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:55:56.393806 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:56:08.174018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:08.173982 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:12.124130 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:12.124098 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:56:12.130133 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:12.130106 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 20:56:26.394517 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:56:26.394440 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:56:42.990839 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:42.990802 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:43.139439 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:43.139376 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:43.342838 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:43.342741 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:43.490942 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:43.490904 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:43.647104 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:43.647070 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:43.799682 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:43.799650 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:49.318815 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:49.318780 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:56:49.319256 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:49.318982 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d608d9aa-dd15-4bde-b8d5-c11870fdea24" containerName="otc-container" containerID="cri-o://245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee" gracePeriod=30 Oct 30 20:56:49.653365 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:49.653325 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:49.798977 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:49.798939 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:50.004909 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:50.004822 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:51.660910 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.660882 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:51.693992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.693953 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs\") pod \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " Oct 30 20:56:51.694156 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.694007 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal\") pod \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " Oct 30 20:56:51.694156 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.694032 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert\") pod \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " Oct 30 20:56:51.694156 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.694060 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z\" (UniqueName: \"kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z\") pod \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " Oct 30 20:56:51.694156 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.694111 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs\") pod \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " Oct 30 20:56:51.694156 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.694127 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs\") pod \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\" (UID: \"d608d9aa-dd15-4bde-b8d5-c11870fdea24\") " Oct 30 20:56:51.694579 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.694548 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal" pod "d608d9aa-dd15-4bde-b8d5-c11870fdea24" (UID: "d608d9aa-dd15-4bde-b8d5-c11870fdea24"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:51.696247 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.696217 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs" pod "d608d9aa-dd15-4bde-b8d5-c11870fdea24" (UID: "d608d9aa-dd15-4bde-b8d5-c11870fdea24"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:51.696374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.696284 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs" pod "d608d9aa-dd15-4bde-b8d5-c11870fdea24" (UID: "d608d9aa-dd15-4bde-b8d5-c11870fdea24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:51.696374 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.696336 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert" pod "d608d9aa-dd15-4bde-b8d5-c11870fdea24" (UID: "d608d9aa-dd15-4bde-b8d5-c11870fdea24"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:51.696662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.696637 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs" pod "d608d9aa-dd15-4bde-b8d5-c11870fdea24" (UID: "d608d9aa-dd15-4bde-b8d5-c11870fdea24"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:51.696726 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.696669 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z" pod "d608d9aa-dd15-4bde-b8d5-c11870fdea24" (UID: "d608d9aa-dd15-4bde-b8d5-c11870fdea24"). InnerVolumeSpecName "kube-api-access-h9j8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:51.795020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.794944 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:51.795020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.794973 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:51.795020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.794983 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:51.795020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.794991 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d608d9aa-dd15-4bde-b8d5-c11870fdea24-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:51.795020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.795000 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d608d9aa-dd15-4bde-b8d5-c11870fdea24-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:51.795020 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:51.795009 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9j8z\" (UniqueName: \"kubernetes.io/projected/d608d9aa-dd15-4bde-b8d5-c11870fdea24-kube-api-access-h9j8z\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:56:52.355123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.355081 4988 generic.go:356] "Generic (PLEG): container finished" podID="d608d9aa-dd15-4bde-b8d5-c11870fdea24" containerID="245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee" exitCode=0 Oct 30 20:56:52.355339 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.355131 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.355339 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.355143 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d608d9aa-dd15-4bde-b8d5-c11870fdea24","Type":"ContainerDied","Data":"245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee"} Oct 30 20:56:52.355339 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.355191 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d608d9aa-dd15-4bde-b8d5-c11870fdea24","Type":"ContainerDied","Data":"30cda92835986f314a171292fcf3af62f7f71075cd32e87046dc0c2fb7a26809"} Oct 30 20:56:52.355339 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.355216 4988 scope.go:122] "RemoveContainer" containerID="245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee" Oct 30 20:56:52.363988 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.363961 4988 scope.go:122] "RemoveContainer" containerID="245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee" Oct 30 20:56:52.364271 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:56:52.364245 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee\": container with ID starting with 245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee not found: ID does not exist" containerID="245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee" Oct 30 20:56:52.364326 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.364283 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee"} err="failed to get container status \"245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee\": rpc error: code = NotFound desc = could not find container \"245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee\": container with ID starting with 245ad5a11bb5059fb837c0037c7d2fc277fd4b21c42e82fd63c284e7d37dbaee not found: ID does not exist" Oct 30 20:56:52.544265 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.544228 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:56:52.558081 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.558038 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:56:52.593224 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.593175 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:56:52.593636 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.593618 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d608d9aa-dd15-4bde-b8d5-c11870fdea24" containerName="otc-container" Oct 30 20:56:52.593702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.593639 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d608d9aa-dd15-4bde-b8d5-c11870fdea24" containerName="otc-container" Oct 30 20:56:52.593748 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.593726 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d608d9aa-dd15-4bde-b8d5-c11870fdea24" containerName="otc-container" Oct 30 20:56:52.598257 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.598234 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.620926 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.620832 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:52.624703 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.624672 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:56:52.628352 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.628332 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.650596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.650564 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:56:52.650782 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.650760 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 20:56:52.650860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.650819 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-6j24j\"" Oct 30 20:56:52.650955 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.650940 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 20:56:52.651066 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.651048 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:56:52.651127 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.651105 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 20:56:52.651127 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.651114 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 20:56:52.653928 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.653907 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 20:56:52.662481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.662451 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:56:52.690327 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.690294 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:56:52.751331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.751293 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 20:56:52.751331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.751300 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:56:52.751623 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.751363 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:56:52.756445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.751887 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:56:52.756445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.751893 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:56:52.756445 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.751912 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:56:52.767357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.767317 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:52.804662 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804620 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.804839 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804734 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.804839 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804796 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74llp\" (UniqueName: \"kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.804839 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804828 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xzj\" (UniqueName: \"kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.804945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804873 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.804945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804920 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.804945 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804940 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.805038 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804973 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.805038 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.804989 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.805038 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.805010 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.805123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.805078 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.805123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.805109 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.805263 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.805242 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.805299 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.805281 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906080 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906035 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906080 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906082 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.906360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906124 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.906360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906141 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906164 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906180 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906328 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74llp\" (UniqueName: \"kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906372 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xzj\" (UniqueName: \"kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906443 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906508 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906534 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906563 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906591 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.906671 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906614 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.907087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906923 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.907087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906949 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.907087 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.906957 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.907473 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.907424 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.907921 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.907885 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.909599 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.909571 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.910055 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.910029 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.910118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.910056 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.910118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.910097 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.910190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.910116 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.910190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.910126 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.910454 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.910438 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:52.967744 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.967704 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74llp\" (UniqueName: \"kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:56:52.967744 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:52.967720 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xzj\" (UniqueName: \"kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj\") pod \"data-science-collector-collector-1\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:53.241603 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:53.241513 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:56:53.619317 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:53.619210 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:56:53.650145 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:56:53.650117 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode930aa07_5b7f_47e3_bf42_8ca380bc9f47.slice/crio-21a58faba0903b588ede1c3c56515c6c12a19119bc435c868bc10b602cb6f168 WatchSource:0}: Error finding container 21a58faba0903b588ede1c3c56515c6c12a19119bc435c868bc10b602cb6f168: Status 404 returned error can't find the container with id 21a58faba0903b588ede1c3c56515c6c12a19119bc435c868bc10b602cb6f168 Oct 30 20:56:53.652282 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:53.652243 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:56:53.652379 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:53.652311 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:56:54.016070 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:54.016034 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d608d9aa-dd15-4bde-b8d5-c11870fdea24" path="/var/lib/kubelet/pods/d608d9aa-dd15-4bde-b8d5-c11870fdea24/volumes" Oct 30 20:56:54.367640 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:54.367604 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e930aa07-5b7f-47e3-bf42-8ca380bc9f47","Type":"ContainerStarted","Data":"b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13"} Oct 30 20:56:54.367812 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:54.367663 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e930aa07-5b7f-47e3-bf42-8ca380bc9f47","Type":"ContainerStarted","Data":"21a58faba0903b588ede1c3c56515c6c12a19119bc435c868bc10b602cb6f168"} Oct 30 20:56:54.485750 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:56:54.485698 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.485684303 podStartE2EDuration="2.485684303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:54.485363175 +0000 UTC m=+1242.592859087" watchObservedRunningTime="2026-10-30 20:56:54.485684303 +0000 UTC m=+1242.593180213" Oct 30 20:56:56.394792 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:56:56.394749 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:57:02.498208 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.498165 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:02.602282 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.602245 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:02.699863 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.699823 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:02.700054 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.699960 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:02.751998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.751916 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:02.892100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.892065 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:02.959215 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:02.959180 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:03.114431 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:03.114320 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:03.597446 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:57:03.597401 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b147876_cdb6_49c0_b267_21050dff86a7.slice/crio-e5bab01ebf4e118b09e5b5f9ea98baff8b0f0246dc1c8f14fb0065f82e0aed7d WatchSource:0}: Error finding container e5bab01ebf4e118b09e5b5f9ea98baff8b0f0246dc1c8f14fb0065f82e0aed7d: Status 404 returned error can't find the container with id e5bab01ebf4e118b09e5b5f9ea98baff8b0f0246dc1c8f14fb0065f82e0aed7d Oct 30 20:57:03.644053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:03.644018 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:57:04.418898 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.418860 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerStarted","Data":"5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae"} Oct 30 20:57:04.419100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.418919 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerStarted","Data":"49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f"} Oct 30 20:57:04.419100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.418938 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerStarted","Data":"7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb"} Oct 30 20:57:04.419100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.418965 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:04.419100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.418981 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerStarted","Data":"e5bab01ebf4e118b09e5b5f9ea98baff8b0f0246dc1c8f14fb0065f82e0aed7d"} Oct 30 20:57:04.419100 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.418999 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:04.421677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.421653 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:04.428324 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.428299 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:04.642215 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:04.642158 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=12.64214098 podStartE2EDuration="12.64214098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:04.590844192 +0000 UTC m=+1252.698340104" watchObservedRunningTime="2026-10-30 20:57:04.64214098 +0000 UTC m=+1252.749636945" Oct 30 20:57:05.423611 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:05.423577 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:08.555205 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:08.555161 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:16.259887 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:16.259848 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:16.460963 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:16.460922 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:16.717952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:16.717912 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:16.863202 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:16.863169 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:17.063806 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:17.063721 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:17.264244 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:17.264206 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:17.405124 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:17.405083 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:17.548888 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:17.548859 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:22.467939 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:22.467902 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:22.468363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:22.468122 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e930aa07-5b7f-47e3-bf42-8ca380bc9f47" containerName="otc-container" containerID="cri-o://b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13" gracePeriod=30 Oct 30 20:57:22.796976 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:22.796890 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:22.945442 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:22.945377 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:23.169095 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:23.169056 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:24.809790 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.809763 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:24.901478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.901404 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs\") pod \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " Oct 30 20:57:24.901677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.901497 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert\") pod \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " Oct 30 20:57:24.901677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.901545 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs\") pod \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " Oct 30 20:57:24.901677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.901596 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj\" (UniqueName: \"kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj\") pod \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " Oct 30 20:57:24.901677 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.901642 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs\") pod \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " Oct 30 20:57:24.901918 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.901713 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal\") pod \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\" (UID: \"e930aa07-5b7f-47e3-bf42-8ca380bc9f47\") " Oct 30 20:57:24.902333 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.902303 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal" pod "e930aa07-5b7f-47e3-bf42-8ca380bc9f47" (UID: "e930aa07-5b7f-47e3-bf42-8ca380bc9f47"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:24.904075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.904037 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs" pod "e930aa07-5b7f-47e3-bf42-8ca380bc9f47" (UID: "e930aa07-5b7f-47e3-bf42-8ca380bc9f47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:24.904346 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.904298 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert" pod "e930aa07-5b7f-47e3-bf42-8ca380bc9f47" (UID: "e930aa07-5b7f-47e3-bf42-8ca380bc9f47"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:24.904497 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.904383 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs" pod "e930aa07-5b7f-47e3-bf42-8ca380bc9f47" (UID: "e930aa07-5b7f-47e3-bf42-8ca380bc9f47"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:24.904587 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.904564 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs" pod "e930aa07-5b7f-47e3-bf42-8ca380bc9f47" (UID: "e930aa07-5b7f-47e3-bf42-8ca380bc9f47"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:24.904695 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:24.904629 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj" pod "e930aa07-5b7f-47e3-bf42-8ca380bc9f47" (UID: "e930aa07-5b7f-47e3-bf42-8ca380bc9f47"). InnerVolumeSpecName "kube-api-access-66xzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:25.003438 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.003355 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.003438 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.003405 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66xzj\" (UniqueName: \"kubernetes.io/projected/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-kube-api-access-66xzj\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.003438 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.003427 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.003438 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.003437 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.003438 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.003446 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.003761 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.003455 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e930aa07-5b7f-47e3-bf42-8ca380bc9f47-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.251367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.251267 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:57:25.252006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.251629 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo" containerID="cri-o://7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb" gracePeriod=30 Oct 30 20:57:25.252006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.251676 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway" containerID="cri-o://49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f" gracePeriod=30 Oct 30 20:57:25.252006 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.251670 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway-opa" containerID="cri-o://5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae" gracePeriod=30 Oct 30 20:57:25.256014 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.255879 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:25.295037 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.295003 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:25.515712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.515623 4988 generic.go:356] "Generic (PLEG): container finished" podID="e930aa07-5b7f-47e3-bf42-8ca380bc9f47" containerID="b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13" exitCode=0 Oct 30 20:57:25.515712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.515669 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.515712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.515701 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e930aa07-5b7f-47e3-bf42-8ca380bc9f47","Type":"ContainerDied","Data":"b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13"} Oct 30 20:57:25.515938 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.515757 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e930aa07-5b7f-47e3-bf42-8ca380bc9f47","Type":"ContainerDied","Data":"21a58faba0903b588ede1c3c56515c6c12a19119bc435c868bc10b602cb6f168"} Oct 30 20:57:25.515938 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.515789 4988 scope.go:122] "RemoveContainer" containerID="b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13" Oct 30 20:57:25.518018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.517998 4988 generic.go:356] "Generic (PLEG): container finished" podID="8b147876-cdb6-49c0-b267-21050dff86a7" containerID="5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae" exitCode=0 Oct 30 20:57:25.518018 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.518018 4988 generic.go:356] "Generic (PLEG): container finished" podID="8b147876-cdb6-49c0-b267-21050dff86a7" containerID="7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb" exitCode=0 Oct 30 20:57:25.518165 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.518057 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerDied","Data":"5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae"} Oct 30 20:57:25.518165 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.518095 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerDied","Data":"7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb"} Oct 30 20:57:25.525772 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.525748 4988 scope.go:122] "RemoveContainer" containerID="b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13" Oct 30 20:57:25.526079 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:25.526058 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13\": container with ID starting with b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13 not found: ID does not exist" containerID="b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13" Oct 30 20:57:25.526145 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.526087 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13"} err="failed to get container status \"b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13\": rpc error: code = NotFound desc = could not find container \"b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13\": container with ID starting with b85d1c9d82e08938a7f6e0c011425d6fdd5a29fcd8541040c6b768cf1ee3de13 not found: ID does not exist" Oct 30 20:57:25.679133 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.679098 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:25.685116 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.685088 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:25.761003 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.760972 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:25.761387 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.761375 4988 cpu_manager.go:414] "containerMap: removing container" podUID="e930aa07-5b7f-47e3-bf42-8ca380bc9f47" containerName="otc-container" Oct 30 20:57:25.761457 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.761388 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e930aa07-5b7f-47e3-bf42-8ca380bc9f47" containerName="otc-container" Oct 30 20:57:25.761494 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.761477 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e930aa07-5b7f-47e3-bf42-8ca380bc9f47" containerName="otc-container" Oct 30 20:57:25.765794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.765753 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.818819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.818719 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:57:25.818819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.818729 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:57:25.818819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.818785 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:57:25.818819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.818794 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:57:25.818819 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.818791 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:57:25.819484 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.819463 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 20:57:25.825119 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.825085 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:25.911235 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.911192 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.911478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.911252 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.911478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.911332 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.911478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.911438 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgj8\" (UniqueName: \"kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.911597 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.911482 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:25.911597 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:25.911500 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.012279 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.012238 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.012279 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.012278 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.012542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.012342 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgj8\" (UniqueName: \"kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.012542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.012381 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.012542 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.012468 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.012696 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.012559 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.013457 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.013429 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.014999 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.014972 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.015235 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.015219 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.015434 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.015398 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.015502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.015481 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.018399 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.018352 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e930aa07-5b7f-47e3-bf42-8ca380bc9f47" path="/var/lib/kubelet/pods/e930aa07-5b7f-47e3-bf42-8ca380bc9f47/volumes" Oct 30 20:57:26.091651 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.091625 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgj8\" (UniqueName: \"kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8\") pod \"data-science-collector-collector-1\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.378067 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.378035 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:26.399671 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:26.399630 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:57:26.522600 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.522566 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:26.525051 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.525025 4988 generic.go:356] "Generic (PLEG): container finished" podID="8b147876-cdb6-49c0-b267-21050dff86a7" containerID="49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f" exitCode=0 Oct 30 20:57:26.525194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.525068 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:26.525194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.525096 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerDied","Data":"49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f"} Oct 30 20:57:26.525194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.525147 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8b147876-cdb6-49c0-b267-21050dff86a7","Type":"ContainerDied","Data":"e5bab01ebf4e118b09e5b5f9ea98baff8b0f0246dc1c8f14fb0065f82e0aed7d"} Oct 30 20:57:26.525194 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.525172 4988 scope.go:122] "RemoveContainer" containerID="5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae" Oct 30 20:57:26.533239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.533221 4988 scope.go:122] "RemoveContainer" containerID="49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f" Oct 30 20:57:26.542392 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.542363 4988 scope.go:122] "RemoveContainer" containerID="7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb" Oct 30 20:57:26.550581 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.550560 4988 scope.go:122] "RemoveContainer" containerID="5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae" Oct 30 20:57:26.550869 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:26.550847 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae\": container with ID starting with 5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae not found: ID does not exist" containerID="5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae" Oct 30 20:57:26.550952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.550882 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae"} err="failed to get container status \"5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae\": rpc error: code = NotFound desc = could not find container \"5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae\": container with ID starting with 5f9d8a3ec8d0f305c35c7856af1e5e08c2ae9941872cbdbcd4d7c53c9ef088ae not found: ID does not exist" Oct 30 20:57:26.550952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.550912 4988 scope.go:122] "RemoveContainer" containerID="49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f" Oct 30 20:57:26.551194 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:26.551175 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f\": container with ID starting with 49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f not found: ID does not exist" containerID="49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f" Oct 30 20:57:26.551232 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.551201 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f"} err="failed to get container status \"49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f\": rpc error: code = NotFound desc = could not find container \"49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f\": container with ID starting with 49807672e6874c535ed944b38c32ee1882d3d3e377f6fc466ab6f240a6b3793f not found: ID does not exist" Oct 30 20:57:26.551232 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.551219 4988 scope.go:122] "RemoveContainer" containerID="7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb" Oct 30 20:57:26.551456 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:26.551436 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb\": container with ID starting with 7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb not found: ID does not exist" containerID="7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb" Oct 30 20:57:26.551529 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.551463 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb"} err="failed to get container status \"7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb\": rpc error: code = NotFound desc = could not find container \"7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb\": container with ID starting with 7d36519cb4566bae9dfd1672cde89346a236b0294ab3418494af934080ee39bb not found: ID does not exist" Oct 30 20:57:26.618112 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618025 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618112 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618065 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618112 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618106 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618427 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618252 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618427 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618286 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp\" (UniqueName: \"kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618427 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618329 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618427 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618364 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618427 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618393 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618700 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618443 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:26.618700 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618462 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf\") pod \"8b147876-cdb6-49c0-b267-21050dff86a7\" (UID: \"8b147876-cdb6-49c0-b267-21050dff86a7\") " Oct 30 20:57:26.618700 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.618492 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:26.619084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.619058 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.619174 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.619088 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.619174 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.619094 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:26.619279 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.619186 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:26.620582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.620553 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "kube-api-access-74llp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:26.620688 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.620623 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:26.620731 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.620712 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:26.620958 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.620930 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:26.621574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.621555 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "8b147876-cdb6-49c0-b267-21050dff86a7" (UID: "8b147876-cdb6-49c0-b267-21050dff86a7"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:57:26.719757 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719717 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.719757 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719750 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.719757 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719762 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.720084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719772 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8b147876-cdb6-49c0-b267-21050dff86a7-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.720084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719782 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8b147876-cdb6-49c0-b267-21050dff86a7-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.720084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719810 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:57:26.720084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.719820 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74llp\" (UniqueName: \"kubernetes.io/projected/8b147876-cdb6-49c0-b267-21050dff86a7-kube-api-access-74llp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:26.737338 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.737310 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:57:26.814109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.814077 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:26.817385 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:57:26.817352 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc49e00_dc57_4a2b_8d10_23d14c5d51c9.slice/crio-a4a56870fe309208c6292a227b6a80408567318fdc845afa2c90e7607b1d6e48 WatchSource:0}: Error finding container a4a56870fe309208c6292a227b6a80408567318fdc845afa2c90e7607b1d6e48: Status 404 returned error can't find the container with id a4a56870fe309208c6292a227b6a80408567318fdc845afa2c90e7607b1d6e48 Oct 30 20:57:26.819688 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.819651 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:57:26.820010 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.819718 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:57:26.820352 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.820332 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 20:57:26.947734 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.947702 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:57:26.952478 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:26.952388 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:57:27.531862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:27.531826 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"adc49e00-dc57-4a2b-8d10-23d14c5d51c9","Type":"ContainerStarted","Data":"2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa"} Oct 30 20:57:27.532042 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:27.531875 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"adc49e00-dc57-4a2b-8d10-23d14c5d51c9","Type":"ContainerStarted","Data":"a4a56870fe309208c6292a227b6a80408567318fdc845afa2c90e7607b1d6e48"} Oct 30 20:57:27.651402 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:27.651348 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.6513324320000002 podStartE2EDuration="2.651332432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:27.650076464 +0000 UTC m=+1275.757572374" watchObservedRunningTime="2026-10-30 20:57:27.651332432 +0000 UTC m=+1275.758828407" Oct 30 20:57:28.016331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:28.016296 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" path="/var/lib/kubelet/pods/8b147876-cdb6-49c0-b267-21050dff86a7/volumes" Oct 30 20:57:29.812159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812125 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q9r7w-cxx2g"] Oct 30 20:57:29.812547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812521 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway" Oct 30 20:57:29.812547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812533 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway" Oct 30 20:57:29.812547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812547 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway-opa" Oct 30 20:57:29.812647 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812552 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway-opa" Oct 30 20:57:29.812647 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812563 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo" Oct 30 20:57:29.812647 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812569 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo" Oct 30 20:57:29.812647 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812629 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway" Oct 30 20:57:29.812647 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812638 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo-gateway-opa" Oct 30 20:57:29.812647 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.812645 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b147876-cdb6-49c0-b267-21050dff86a7" containerName="tempo" Oct 30 20:57:29.815815 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.815797 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:29.873935 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.873898 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q9r7w-cxx2g"] Oct 30 20:57:29.947155 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.947119 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:29.947350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.947196 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5nw\" (UniqueName: \"kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:29.947350 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.947306 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:29.947473 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:29.947356 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.048669 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.048635 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5nw\" (UniqueName: \"kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.048842 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.048689 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.048842 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.048715 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.048842 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.048737 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.049092 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.049075 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.049389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.049366 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.049556 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.049536 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.110282 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.110203 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5nw\" (UniqueName: \"kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw\") pod \"periodic-gathering-q9r7w-cxx2g\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.128103 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.128067 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:57:30.502629 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.502593 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q9r7w-cxx2g"] Oct 30 20:57:30.506351 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:57:30.506316 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed73e4c_ac36_4123_bbd6_4a2cdda6f26c.slice/crio-77de5f51c3ab9698a0ca5ec9737ef5f3dc323c002d933ec45f03366cb7c45aab WatchSource:0}: Error finding container 77de5f51c3ab9698a0ca5ec9737ef5f3dc323c002d933ec45f03366cb7c45aab: Status 404 returned error can't find the container with id 77de5f51c3ab9698a0ca5ec9737ef5f3dc323c002d933ec45f03366cb7c45aab Oct 30 20:57:30.548802 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:30.548770 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" event={"ID":"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c","Type":"ContainerStarted","Data":"77de5f51c3ab9698a0ca5ec9737ef5f3dc323c002d933ec45f03366cb7c45aab"} Oct 30 20:57:31.556113 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:31.556069 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" event={"ID":"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c","Type":"ContainerStarted","Data":"b5e3fb2e5455016552bdb0d6cd1b78568598173fccc5f19d2351319439e31aca"} Oct 30 20:57:31.672330 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:31.672272 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" podStartSLOduration=2.67225527 podStartE2EDuration="2.67225527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:31.671543777 +0000 UTC m=+1279.779039689" watchObservedRunningTime="2026-10-30 20:57:31.67225527 +0000 UTC m=+1279.779751179" Oct 30 20:57:49.473688 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:49.473618 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:49.617972 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:49.617931 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:49.772233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:49.772137 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:49.923998 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:49.923956 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:50.128596 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:50.128502 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:50.281084 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:50.281045 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:54.304749 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:54.304666 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:55.596523 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:55.596487 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:55.596967 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:55.596696 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="adc49e00-dc57-4a2b-8d10-23d14c5d51c9" containerName="otc-container" containerID="cri-o://2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa" gracePeriod=30 Oct 30 20:57:55.967913 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:55.967867 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.125650 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:56.125617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.322208 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:56.322110 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.395071 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:56.395033 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:57:57.968848 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:57.968817 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:58.092711 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.092619 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs\") pod \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " Oct 30 20:57:58.092711 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.092664 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8\" (UniqueName: \"kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8\") pod \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " Oct 30 20:57:58.092711 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.092687 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs\") pod \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " Oct 30 20:57:58.093003 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.092748 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert\") pod \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " Oct 30 20:57:58.093003 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.092807 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal\") pod \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " Oct 30 20:57:58.093003 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.092832 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs\") pod \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\" (UID: \"adc49e00-dc57-4a2b-8d10-23d14c5d51c9\") " Oct 30 20:57:58.093493 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.093450 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal" pod "adc49e00-dc57-4a2b-8d10-23d14c5d51c9" (UID: "adc49e00-dc57-4a2b-8d10-23d14c5d51c9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:58.095637 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.095591 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert" pod "adc49e00-dc57-4a2b-8d10-23d14c5d51c9" (UID: "adc49e00-dc57-4a2b-8d10-23d14c5d51c9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:58.095993 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.095900 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs" pod "adc49e00-dc57-4a2b-8d10-23d14c5d51c9" (UID: "adc49e00-dc57-4a2b-8d10-23d14c5d51c9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:58.095993 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.095954 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8" pod "adc49e00-dc57-4a2b-8d10-23d14c5d51c9" (UID: "adc49e00-dc57-4a2b-8d10-23d14c5d51c9"). InnerVolumeSpecName "kube-api-access-5bgj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:58.096218 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.096192 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs" pod "adc49e00-dc57-4a2b-8d10-23d14c5d51c9" (UID: "adc49e00-dc57-4a2b-8d10-23d14c5d51c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:58.101892 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.101859 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs" pod "adc49e00-dc57-4a2b-8d10-23d14c5d51c9" (UID: "adc49e00-dc57-4a2b-8d10-23d14c5d51c9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:58.194487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.194327 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.194487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.194371 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgj8\" (UniqueName: \"kubernetes.io/projected/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-kube-api-access-5bgj8\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.194487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.194390 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.194487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.194422 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.194487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.194441 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.194487 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.194455 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/adc49e00-dc57-4a2b-8d10-23d14c5d51c9-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.623652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.623604 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:57:58.624070 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.624054 4988 cpu_manager.go:414] "containerMap: removing container" podUID="adc49e00-dc57-4a2b-8d10-23d14c5d51c9" containerName="otc-container" Oct 30 20:57:58.624132 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.624071 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc49e00-dc57-4a2b-8d10-23d14c5d51c9" containerName="otc-container" Oct 30 20:57:58.624168 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.624136 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc49e00-dc57-4a2b-8d10-23d14c5d51c9" containerName="otc-container" Oct 30 20:57:58.629332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.629296 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.683745 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.683708 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 20:57:58.683745 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.683725 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:57:58.684003 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.683800 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-5gwlb\"" Oct 30 20:57:58.684209 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.684179 4988 generic.go:356] "Generic (PLEG): container finished" podID="adc49e00-dc57-4a2b-8d10-23d14c5d51c9" containerID="2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa" exitCode=0 Oct 30 20:57:58.684335 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.684266 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"adc49e00-dc57-4a2b-8d10-23d14c5d51c9","Type":"ContainerDied","Data":"2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa"} Oct 30 20:57:58.684335 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.684296 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:58.684335 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.684324 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"adc49e00-dc57-4a2b-8d10-23d14c5d51c9","Type":"ContainerDied","Data":"a4a56870fe309208c6292a227b6a80408567318fdc845afa2c90e7607b1d6e48"} Oct 30 20:57:58.684492 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.684354 4988 scope.go:122] "RemoveContainer" containerID="2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa" Oct 30 20:57:58.695182 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.695157 4988 scope.go:122] "RemoveContainer" containerID="2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa" Oct 30 20:57:58.695747 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:58.695674 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa\": container with ID starting with 2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa not found: ID does not exist" containerID="2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa" Oct 30 20:57:58.695831 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.695758 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa"} err="failed to get container status \"2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa\": rpc error: code = NotFound desc = could not find container \"2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa\": container with ID starting with 2ed3dd7dfa475cf2356e6f191376038bf31a959cd374e31aa7142ba18397cffa not found: ID does not exist" Oct 30 20:57:58.732794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.732745 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 20:57:58.732794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.732754 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 20:57:58.732794 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.732799 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 20:57:58.733054 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.732827 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 20:57:58.733054 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.732926 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 20:57:58.743742 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.743708 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:57:58.745077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.745050 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:58.798992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.798954 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.798992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.798994 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.799239 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.799018 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.799294 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.799227 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.799353 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.799334 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.799437 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.799362 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jvh\" (UniqueName: \"kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.799495 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.799462 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.799554 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.799505 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905726 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905688 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905726 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905732 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jvh\" (UniqueName: \"kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905780 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905806 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905895 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905939 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.905995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.905968 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.906259 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:58.906006 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:57:58.906259 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:57:58.906107 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert podName:8f0b0c5f-5266-49aa-873b-c2d6569b00b1 nodeName:}" failed. No retries permitted until 2026-10-30 20:57:59.406081763 +0000 UTC m=+1307.513577666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 30 20:57:58.906259 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.906011 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.906707 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.906683 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.907233 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.907204 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.907385 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.907358 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.907639 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.907619 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.907831 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.907817 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:58.909105 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.909073 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.909225 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.909208 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:58.909597 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.909578 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:58.967814 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:58.967782 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jvh\" (UniqueName: \"kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:59.020363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.020323 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:59.026304 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.026274 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.094690 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.094654 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:57:59.094866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.094733 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:57:59.094866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.094764 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:57:59.094866 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.094819 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:57:59.094995 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.094900 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:57:59.107511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.107478 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.107702 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.107568 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.107758 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.107693 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.107758 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.107742 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j7cc\" (UniqueName: \"kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.107834 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.107803 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.107873 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.107836 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.117077 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.117048 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:59.137993 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.137963 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 20:57:59.208363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.208270 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.208363 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.208359 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.208628 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.208391 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.208628 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.208563 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j7cc\" (UniqueName: \"kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.208742 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.208638 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.208742 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.208675 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.209620 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.209590 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.211054 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.211026 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.211291 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.211270 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.211694 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.211666 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.211993 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.211973 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.250439 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.250224 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:59.274547 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.274514 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j7cc\" (UniqueName: \"kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc\") pod \"data-science-collector-collector-1\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.340402 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.340363 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:57:59.413515 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.412994 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:59.417751 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.417671 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:57:59.816419 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.816366 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:57:59.820098 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:57:59.820066 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fbf21c_c234_4b4d_81b5_48a855487e6a.slice/crio-fd8ac872c1a027c0e3a6f26e7a38feabe8fdbda758e4f75826806ba610d5d0a0 WatchSource:0}: Error finding container fd8ac872c1a027c0e3a6f26e7a38feabe8fdbda758e4f75826806ba610d5d0a0: Status 404 returned error can't find the container with id fd8ac872c1a027c0e3a6f26e7a38feabe8fdbda758e4f75826806ba610d5d0a0 Oct 30 20:57:59.822426 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.822376 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:57:59.822535 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:57:59.822473 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:58:00.019315 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:00.019272 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc49e00-dc57-4a2b-8d10-23d14c5d51c9" path="/var/lib/kubelet/pods/adc49e00-dc57-4a2b-8d10-23d14c5d51c9/volumes" Oct 30 20:58:00.701032 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:00.700997 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"95fbf21c-c234-4b4d-81b5-48a855487e6a","Type":"ContainerStarted","Data":"69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb"} Oct 30 20:58:00.701496 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:00.701051 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"95fbf21c-c234-4b4d-81b5-48a855487e6a","Type":"ContainerStarted","Data":"fd8ac872c1a027c0e3a6f26e7a38feabe8fdbda758e4f75826806ba610d5d0a0"} Oct 30 20:58:00.820114 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:00.820051 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.820031577 podStartE2EDuration="2.820031577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:00.818862932 +0000 UTC m=+1308.926358848" watchObservedRunningTime="2026-10-30 20:58:00.820031577 +0000 UTC m=+1308.927527488" Oct 30 20:58:04.374627 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.374588 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.478834 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.478792 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.576652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.576609 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.576859 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.576759 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.628559 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.628469 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.768901 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.768858 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.834483 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.834446 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:04.951574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:04.951525 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:05.437492 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.437275 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:58:05.441332 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:58:05.441299 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0b0c5f_5266_49aa_873b_c2d6569b00b1.slice/crio-f30bacbe00dd00a82e101db556cea2daff62ff1b331816e7d971d5e3f8e3ce6a WatchSource:0}: Error finding container f30bacbe00dd00a82e101db556cea2daff62ff1b331816e7d971d5e3f8e3ce6a: Status 404 returned error can't find the container with id f30bacbe00dd00a82e101db556cea2daff62ff1b331816e7d971d5e3f8e3ce6a Oct 30 20:58:05.728545 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.728455 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerStarted","Data":"89ef57cd39945ad36e8cafbb4cf09fda38bc1363f97f34a61886975987b1fc3b"} Oct 30 20:58:05.728545 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.728517 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerStarted","Data":"d0af5598ed4a3d5eb7d0ad8107b83ba0923c713642e2cefbc5adcdf7d4bfa1dc"} Oct 30 20:58:05.728545 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.728543 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerStarted","Data":"60137b931a17a20f3f6f0795aa8169c8c87608f5891372548c1d09989e78f896"} Oct 30 20:58:05.728841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.728568 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerStarted","Data":"f30bacbe00dd00a82e101db556cea2daff62ff1b331816e7d971d5e3f8e3ce6a"} Oct 30 20:58:05.728841 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.728684 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:05.731503 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.731481 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:05.852331 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:05.852267 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.852252543 podStartE2EDuration="7.852252543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:05.851339162 +0000 UTC m=+1313.958835078" watchObservedRunningTime="2026-10-30 20:58:05.852252543 +0000 UTC m=+1313.959748455" Oct 30 20:58:06.733827 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:06.733786 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:06.734271 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:06.733887 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:06.741821 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:06.741790 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:06.765389 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:06.765358 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:07.004690 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:07.004597 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:08.840474 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:08.840435 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.095754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:23.095704 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.294480 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:23.294441 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.442082 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:23.442044 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.592965 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:23.592927 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.739815 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:23.739719 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.887248 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:23.887200 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:26.396295 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:58:26.396255 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:58:28.745384 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:28.745344 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:28.852511 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:28.852331 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:58:28.852741 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:28.852653 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="95fbf21c-c234-4b4d-81b5-48a855487e6a" containerName="otc-container" containerID="cri-o://69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb" gracePeriod=30 Oct 30 20:58:29.208220 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:29.208171 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:29.371205 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:29.371166 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:29.564268 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:29.564175 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:31.208696 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.208663 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:31.316720 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.316623 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal\") pod \"95fbf21c-c234-4b4d-81b5-48a855487e6a\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " Oct 30 20:58:31.316720 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.316678 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert\") pod \"95fbf21c-c234-4b4d-81b5-48a855487e6a\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " Oct 30 20:58:31.316720 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.316699 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs\") pod \"95fbf21c-c234-4b4d-81b5-48a855487e6a\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " Oct 30 20:58:31.317007 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.316728 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc\" (UniqueName: \"kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc\") pod \"95fbf21c-c234-4b4d-81b5-48a855487e6a\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " Oct 30 20:58:31.317007 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.316764 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs\") pod \"95fbf21c-c234-4b4d-81b5-48a855487e6a\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " Oct 30 20:58:31.317007 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.316923 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs\") pod \"95fbf21c-c234-4b4d-81b5-48a855487e6a\" (UID: \"95fbf21c-c234-4b4d-81b5-48a855487e6a\") " Oct 30 20:58:31.317173 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.317144 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal" pod "95fbf21c-c234-4b4d-81b5-48a855487e6a" (UID: "95fbf21c-c234-4b4d-81b5-48a855487e6a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:31.317365 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.317343 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95fbf21c-c234-4b4d-81b5-48a855487e6a-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.319127 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.319092 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs" pod "95fbf21c-c234-4b4d-81b5-48a855487e6a" (UID: "95fbf21c-c234-4b4d-81b5-48a855487e6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:31.319258 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.319236 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs" pod "95fbf21c-c234-4b4d-81b5-48a855487e6a" (UID: "95fbf21c-c234-4b4d-81b5-48a855487e6a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:31.319326 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.319305 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc" pod "95fbf21c-c234-4b4d-81b5-48a855487e6a" (UID: "95fbf21c-c234-4b4d-81b5-48a855487e6a"). InnerVolumeSpecName "kube-api-access-5j7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.319431 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.319394 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs" pod "95fbf21c-c234-4b4d-81b5-48a855487e6a" (UID: "95fbf21c-c234-4b4d-81b5-48a855487e6a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:31.319492 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.319426 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert" pod "95fbf21c-c234-4b4d-81b5-48a855487e6a" (UID: "95fbf21c-c234-4b4d-81b5-48a855487e6a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:31.418571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.418531 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.418571 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.418569 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.418807 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.418584 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j7cc\" (UniqueName: \"kubernetes.io/projected/95fbf21c-c234-4b4d-81b5-48a855487e6a-kube-api-access-5j7cc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.418807 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.418599 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.418807 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.418612 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/95fbf21c-c234-4b4d-81b5-48a855487e6a-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.715672 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.715635 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:58:31.715939 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.715883 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo" containerID="cri-o://60137b931a17a20f3f6f0795aa8169c8c87608f5891372548c1d09989e78f896" gracePeriod=30 Oct 30 20:58:31.716099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.715931 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway-opa" containerID="cri-o://89ef57cd39945ad36e8cafbb4cf09fda38bc1363f97f34a61886975987b1fc3b" gracePeriod=30 Oct 30 20:58:31.716099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.715990 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway" containerID="cri-o://d0af5598ed4a3d5eb7d0ad8107b83ba0923c713642e2cefbc5adcdf7d4bfa1dc" gracePeriod=30 Oct 30 20:58:31.717498 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.717449 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:31.728678 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.728647 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:31.739358 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.739317 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:31.855312 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.855275 4988 generic.go:356] "Generic (PLEG): container finished" podID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerID="89ef57cd39945ad36e8cafbb4cf09fda38bc1363f97f34a61886975987b1fc3b" exitCode=0 Oct 30 20:58:31.855536 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.855343 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerDied","Data":"89ef57cd39945ad36e8cafbb4cf09fda38bc1363f97f34a61886975987b1fc3b"} Oct 30 20:58:31.856856 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.856828 4988 generic.go:356] "Generic (PLEG): container finished" podID="95fbf21c-c234-4b4d-81b5-48a855487e6a" containerID="69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb" exitCode=0 Oct 30 20:58:31.856948 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.856895 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"95fbf21c-c234-4b4d-81b5-48a855487e6a","Type":"ContainerDied","Data":"69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb"} Oct 30 20:58:31.856992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.856951 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:31.856992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.856968 4988 scope.go:122] "RemoveContainer" containerID="69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb" Oct 30 20:58:31.857085 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.856954 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"95fbf21c-c234-4b4d-81b5-48a855487e6a","Type":"ContainerDied","Data":"fd8ac872c1a027c0e3a6f26e7a38feabe8fdbda758e4f75826806ba610d5d0a0"} Oct 30 20:58:31.866124 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.866092 4988 scope.go:122] "RemoveContainer" containerID="69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb" Oct 30 20:58:31.866588 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:58:31.866562 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb\": container with ID starting with 69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb not found: ID does not exist" containerID="69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb" Oct 30 20:58:31.866686 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:31.866598 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb"} err="failed to get container status \"69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb\": rpc error: code = NotFound desc = could not find container \"69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb\": container with ID starting with 69ad35a75ac8ca33e0293c6e0d8c018fb790c613bdde29695b5708807d82bafb not found: ID does not exist" Oct 30 20:58:32.020856 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.020820 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:58:32.024916 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.024883 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:58:32.099238 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.099202 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:58:32.099684 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.099668 4988 cpu_manager.go:414] "containerMap: removing container" podUID="95fbf21c-c234-4b4d-81b5-48a855487e6a" containerName="otc-container" Oct 30 20:58:32.099736 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.099686 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fbf21c-c234-4b4d-81b5-48a855487e6a" containerName="otc-container" Oct 30 20:58:32.099770 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.099759 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fbf21c-c234-4b4d-81b5-48a855487e6a" containerName="otc-container" Oct 30 20:58:32.102940 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.102919 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.158310 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.158273 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:58:32.158567 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.158489 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:58:32.158567 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.158487 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:58:32.158567 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.158487 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 20:58:32.158567 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.158494 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:58:32.158567 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.158550 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:58:32.170189 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.170150 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:58:32.225249 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.225198 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.225679 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.225262 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzsr\" (UniqueName: \"kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.225679 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.225436 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.225679 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.225474 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.225679 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.225505 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.225679 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.225579 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.327000 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.326918 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.327000 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.326966 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.327000 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.326982 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.327213 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.327100 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.327213 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.327164 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.327315 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.327214 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzsr\" (UniqueName: \"kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.328038 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.328008 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.329731 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.329699 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.329731 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.329726 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.329906 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.329718 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.329906 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.329764 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.388534 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.388494 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzsr\" (UniqueName: \"kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr\") pod \"data-science-collector-collector-1\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.416076 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.416033 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:58:32.796869 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.796833 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:58:32.841965 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:58:32.841928 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e29659f_438f_40e6_b2cb_1b3b48c4403c.slice/crio-70152eed670b5b103f55f759cbd866f10fdac7cbc6178c9b82885f2ae97e348a WatchSource:0}: Error finding container 70152eed670b5b103f55f759cbd866f10fdac7cbc6178c9b82885f2ae97e348a: Status 404 returned error can't find the container with id 70152eed670b5b103f55f759cbd866f10fdac7cbc6178c9b82885f2ae97e348a Oct 30 20:58:32.844174 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.844140 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:58:32.844260 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.844205 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:58:32.865651 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.865612 4988 generic.go:356] "Generic (PLEG): container finished" podID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerID="d0af5598ed4a3d5eb7d0ad8107b83ba0923c713642e2cefbc5adcdf7d4bfa1dc" exitCode=0 Oct 30 20:58:32.865651 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.865646 4988 generic.go:356] "Generic (PLEG): container finished" podID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerID="60137b931a17a20f3f6f0795aa8169c8c87608f5891372548c1d09989e78f896" exitCode=0 Oct 30 20:58:32.865844 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.865678 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerDied","Data":"d0af5598ed4a3d5eb7d0ad8107b83ba0923c713642e2cefbc5adcdf7d4bfa1dc"} Oct 30 20:58:32.865844 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.865734 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerDied","Data":"60137b931a17a20f3f6f0795aa8169c8c87608f5891372548c1d09989e78f896"} Oct 30 20:58:32.868248 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.868155 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9e29659f-438f-40e6-b2cb-1b3b48c4403c","Type":"ContainerStarted","Data":"70152eed670b5b103f55f759cbd866f10fdac7cbc6178c9b82885f2ae97e348a"} Oct 30 20:58:32.965152 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:32.965124 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:33.033044 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.032995 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033056 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033103 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh\" (UniqueName: \"kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033131 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033189 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033266 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033220 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033274 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033368 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033525 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:33.033574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033539 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:33.033574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033531 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\" (UID: \"8f0b0c5f-5266-49aa-873b-c2d6569b00b1\") " Oct 30 20:58:33.033997 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033969 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.034108 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.034003 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.034108 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033973 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:33.034108 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.033980 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:33.036469 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.036440 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "kube-api-access-c2jvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:33.036626 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.036519 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:33.036698 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.036623 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:33.037004 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.036979 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:33.037308 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.037291 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "8f0b0c5f-5266-49aa-873b-c2d6569b00b1" (UID: "8f0b0c5f-5266-49aa-873b-c2d6569b00b1"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 20:58:33.135279 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135235 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.135502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135299 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 20:58:33.135502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135318 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.135502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135334 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jvh\" (UniqueName: \"kubernetes.io/projected/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-kube-api-access-c2jvh\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.135502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135348 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.135502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135364 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.135502 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.135377 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8f0b0c5f-5266-49aa-873b-c2d6569b00b1-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:58:33.151190 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.151163 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 20:58:33.236590 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.236547 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 20:58:33.876043 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.876013 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 20:58:33.876253 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.876012 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8f0b0c5f-5266-49aa-873b-c2d6569b00b1","Type":"ContainerDied","Data":"f30bacbe00dd00a82e101db556cea2daff62ff1b331816e7d971d5e3f8e3ce6a"} Oct 30 20:58:33.876253 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.876153 4988 scope.go:122] "RemoveContainer" containerID="89ef57cd39945ad36e8cafbb4cf09fda38bc1363f97f34a61886975987b1fc3b" Oct 30 20:58:33.877885 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.877861 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9e29659f-438f-40e6-b2cb-1b3b48c4403c","Type":"ContainerStarted","Data":"ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8"} Oct 30 20:58:33.885545 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.885519 4988 scope.go:122] "RemoveContainer" containerID="d0af5598ed4a3d5eb7d0ad8107b83ba0923c713642e2cefbc5adcdf7d4bfa1dc" Oct 30 20:58:33.895966 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:33.895944 4988 scope.go:122] "RemoveContainer" containerID="60137b931a17a20f3f6f0795aa8169c8c87608f5891372548c1d09989e78f896" Oct 30 20:58:34.021759 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:34.021720 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fbf21c-c234-4b4d-81b5-48a855487e6a" path="/var/lib/kubelet/pods/95fbf21c-c234-4b4d-81b5-48a855487e6a/volumes" Oct 30 20:58:34.091481 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:34.091443 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:58:34.141365 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:34.141328 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 20:58:34.218719 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:34.218661 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.218647542 podStartE2EDuration="2.218647542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:34.217705403 +0000 UTC m=+1342.325201316" watchObservedRunningTime="2026-10-30 20:58:34.218647542 +0000 UTC m=+1342.326143453" Oct 30 20:58:36.015598 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:36.015542 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" path="/var/lib/kubelet/pods/8f0b0c5f-5266-49aa-873b-c2d6569b00b1/volumes" Oct 30 20:58:47.571697 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:47.571660 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:47.768247 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:58:47.768207 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:56.395152 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:58:56.395099 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:59:06.029203 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:06.029166 4988 generic.go:356] "Generic (PLEG): container finished" podID="4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" containerID="b5e3fb2e5455016552bdb0d6cd1b78568598173fccc5f19d2351319439e31aca" exitCode=0 Oct 30 20:59:06.029623 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:06.029242 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" event={"ID":"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c","Type":"ContainerDied","Data":"b5e3fb2e5455016552bdb0d6cd1b78568598173fccc5f19d2351319439e31aca"} Oct 30 20:59:07.172360 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.172329 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:59:07.342505 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342388 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle\") pod \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " Oct 30 20:59:07.342505 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342496 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path\") pod \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " Oct 30 20:59:07.342714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342520 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle\") pod \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " Oct 30 20:59:07.342714 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342596 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw\" (UniqueName: \"kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw\") pod \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\" (UID: \"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c\") " Oct 30 20:59:07.342873 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342840 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path" pod "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" (UID: "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:07.342873 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342865 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle" pod "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" (UID: "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:07.342992 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.342967 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle" pod "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" (UID: "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:07.344825 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.344796 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw" pod "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" (UID: "4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c"). InnerVolumeSpecName "kube-api-access-7r5nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:07.443357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.443318 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5nw\" (UniqueName: \"kubernetes.io/projected/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-kube-api-access-7r5nw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:07.443357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.443353 4988 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-service-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:07.443357 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.443363 4988 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-archives-path\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:07.443644 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.443372 4988 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c-trusted-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:07.801712 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:07.801672 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.040159 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.040124 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" Oct 30 20:59:08.040367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.040155 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q9r7w-cxx2g" event={"ID":"4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c","Type":"ContainerDied","Data":"77de5f51c3ab9698a0ca5ec9737ef5f3dc323c002d933ec45f03366cb7c45aab"} Oct 30 20:59:08.040367 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.040207 4988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77de5f51c3ab9698a0ca5ec9737ef5f3dc323c002d933ec45f03366cb7c45aab" Oct 30 20:59:08.043053 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.043007 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.251099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.251057 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.398768 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.398729 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.542754 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.542641 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.686977 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:08.686919 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:09.061075 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:09.061032 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:09.215644 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:09.215596 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:09.359713 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:09.359625 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:13.526201 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:13.526163 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:13.526630 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:13.526379 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="9e29659f-438f-40e6-b2cb-1b3b48c4403c" containerName="otc-container" containerID="cri-o://ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8" gracePeriod=30 Oct 30 20:59:13.876452 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:13.876346 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:14.025532 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:14.025473 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:14.232441 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:14.232383 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:15.882632 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.882596 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:15.901667 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.901631 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr\" (UniqueName: \"kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr\") pod \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " Oct 30 20:59:15.901667 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.901670 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs\") pod \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " Oct 30 20:59:15.901922 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.901714 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs\") pod \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " Oct 30 20:59:15.901922 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.901734 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs\") pod \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " Oct 30 20:59:15.901922 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.901763 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal\") pod \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " Oct 30 20:59:15.901922 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.901780 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert\") pod \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\" (UID: \"9e29659f-438f-40e6-b2cb-1b3b48c4403c\") " Oct 30 20:59:15.902265 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.902224 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal" pod "9e29659f-438f-40e6-b2cb-1b3b48c4403c" (UID: "9e29659f-438f-40e6-b2cb-1b3b48c4403c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:15.904212 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.904177 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs" pod "9e29659f-438f-40e6-b2cb-1b3b48c4403c" (UID: "9e29659f-438f-40e6-b2cb-1b3b48c4403c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:15.904332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.904223 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs" pod "9e29659f-438f-40e6-b2cb-1b3b48c4403c" (UID: "9e29659f-438f-40e6-b2cb-1b3b48c4403c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:15.904332 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.904226 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr" pod "9e29659f-438f-40e6-b2cb-1b3b48c4403c" (UID: "9e29659f-438f-40e6-b2cb-1b3b48c4403c"). InnerVolumeSpecName "kube-api-access-9dzsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:15.904486 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.904461 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert" pod "9e29659f-438f-40e6-b2cb-1b3b48c4403c" (UID: "9e29659f-438f-40e6-b2cb-1b3b48c4403c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:15.904541 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:15.904499 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs" pod "9e29659f-438f-40e6-b2cb-1b3b48c4403c" (UID: "9e29659f-438f-40e6-b2cb-1b3b48c4403c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:16.003221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.003142 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzsr\" (UniqueName: \"kubernetes.io/projected/9e29659f-438f-40e6-b2cb-1b3b48c4403c-kube-api-access-9dzsr\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.003221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.003177 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.003221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.003192 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.003221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.003201 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.003221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.003213 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e29659f-438f-40e6-b2cb-1b3b48c4403c-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.003221 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.003225 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9e29659f-438f-40e6-b2cb-1b3b48c4403c-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.080716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.080674 4988 generic.go:356] "Generic (PLEG): container finished" podID="9e29659f-438f-40e6-b2cb-1b3b48c4403c" containerID="ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8" exitCode=0 Oct 30 20:59:16.080905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.080723 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.080905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.080764 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9e29659f-438f-40e6-b2cb-1b3b48c4403c","Type":"ContainerDied","Data":"ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8"} Oct 30 20:59:16.080905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.080818 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9e29659f-438f-40e6-b2cb-1b3b48c4403c","Type":"ContainerDied","Data":"70152eed670b5b103f55f759cbd866f10fdac7cbc6178c9b82885f2ae97e348a"} Oct 30 20:59:16.080905 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.080840 4988 scope.go:122] "RemoveContainer" containerID="ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8" Oct 30 20:59:16.090102 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.090080 4988 scope.go:122] "RemoveContainer" containerID="ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8" Oct 30 20:59:16.090472 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:59:16.090452 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8\": container with ID starting with ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8 not found: ID does not exist" containerID="ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8" Oct 30 20:59:16.090520 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.090484 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8"} err="failed to get container status \"ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8\": rpc error: code = NotFound desc = could not find container \"ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8\": container with ID starting with ae84a4981c5387d85018b1584f76443b18392ae294cb8ae586ffc0c7abadf0b8 not found: ID does not exist" Oct 30 20:59:16.244557 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.244524 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:16.249051 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.249018 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:16.324101 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324015 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:16.324436 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324405 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo" Oct 30 20:59:16.324485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324435 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo" Oct 30 20:59:16.324485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324450 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway-opa" Oct 30 20:59:16.324485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324456 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway-opa" Oct 30 20:59:16.324485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324465 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway" Oct 30 20:59:16.324485 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324470 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324488 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9e29659f-438f-40e6-b2cb-1b3b48c4403c" containerName="otc-container" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324493 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e29659f-438f-40e6-b2cb-1b3b48c4403c" containerName="otc-container" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324502 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" containerName="insights-gathering" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324507 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" containerName="insights-gathering" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324565 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e29659f-438f-40e6-b2cb-1b3b48c4403c" containerName="otc-container" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324573 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed73e4c-ac36-4123-bbd6-4a2cdda6f26c" containerName="insights-gathering" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324582 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324588 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway-opa" Oct 30 20:59:16.324635 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.324593 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0b0c5f-5266-49aa-873b-c2d6569b00b1" containerName="tempo-gateway" Oct 30 20:59:16.328872 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.328850 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.381500 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.381467 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:59:16.381500 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.381531 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:59:16.381845 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.381571 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\"" Oct 30 20:59:16.381845 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.381584 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:59:16.381845 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.381592 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:59:16.381845 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.381580 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:59:16.387862 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.387830 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:16.406297 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.406255 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbz7\" (UniqueName: \"kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.406488 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.406323 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.406533 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.406481 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.406574 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.406539 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.406616 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.406600 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.406673 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.406645 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.507817 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.507775 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.507817 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.507817 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.508049 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.507845 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.508049 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.507895 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbz7\" (UniqueName: \"kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.508049 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.507931 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.508049 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.507967 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.508589 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.508564 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.510582 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.510556 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.510933 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.510909 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.510984 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.510921 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.511045 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.511030 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.569404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.569368 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbz7\" (UniqueName: \"kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7\") pod \"data-science-collector-collector-1\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:16.642288 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:16.642238 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:17.022814 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:17.022724 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:17.026394 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:59:17.026359 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995e4665_61b8_494c_8cf4_ecaf186a4922.slice/crio-a20c5dc0bab46c49c53c95b5c3e184516aeec2060114307c798f83bb8cad8fc8 WatchSource:0}: Error finding container a20c5dc0bab46c49c53c95b5c3e184516aeec2060114307c798f83bb8cad8fc8: Status 404 returned error can't find the container with id a20c5dc0bab46c49c53c95b5c3e184516aeec2060114307c798f83bb8cad8fc8 Oct 30 20:59:17.028860 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:17.028815 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:59:17.029002 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:17.028903 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:59:17.087778 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:17.087734 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"995e4665-61b8-494c-8cf4-ecaf186a4922","Type":"ContainerStarted","Data":"a20c5dc0bab46c49c53c95b5c3e184516aeec2060114307c798f83bb8cad8fc8"} Oct 30 20:59:18.016632 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:18.016594 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e29659f-438f-40e6-b2cb-1b3b48c4403c" path="/var/lib/kubelet/pods/9e29659f-438f-40e6-b2cb-1b3b48c4403c/volumes" Oct 30 20:59:18.096308 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:18.096267 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"995e4665-61b8-494c-8cf4-ecaf186a4922","Type":"ContainerStarted","Data":"af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f"} Oct 30 20:59:26.395922 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:59:26.395831 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 20:59:39.067123 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.067077 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.217329 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.217291 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.363821 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.363718 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.512186 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.512149 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.666074 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.666033 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.812290 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.812254 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.965268 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:39.965179 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:44.861629 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:44.861560 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=28.861541331 podStartE2EDuration="28.861541331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:18.263395692 +0000 UTC m=+1386.370891604" watchObservedRunningTime="2026-10-30 20:59:44.861541331 +0000 UTC m=+1412.969037244" Oct 30 20:59:44.863520 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:44.863495 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:44.863729 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:44.863708 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="995e4665-61b8-494c-8cf4-ecaf186a4922" containerName="otc-container" containerID="cri-o://af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f" gracePeriod=30 Oct 30 20:59:45.198161 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:45.198124 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:45.341957 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:45.341921 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:45.565094 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:45.565002 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:46.124405 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.124373 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.239609 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.239515 4988 generic.go:356] "Generic (PLEG): container finished" podID="995e4665-61b8-494c-8cf4-ecaf186a4922" containerID="af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f" exitCode=0 Oct 30 20:59:46.239609 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.239566 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.239799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.239610 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"995e4665-61b8-494c-8cf4-ecaf186a4922","Type":"ContainerDied","Data":"af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f"} Oct 30 20:59:46.239799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.239668 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"995e4665-61b8-494c-8cf4-ecaf186a4922","Type":"ContainerDied","Data":"a20c5dc0bab46c49c53c95b5c3e184516aeec2060114307c798f83bb8cad8fc8"} Oct 30 20:59:46.239799 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.239700 4988 scope.go:122] "RemoveContainer" containerID="af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f" Oct 30 20:59:46.249143 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.249118 4988 scope.go:122] "RemoveContainer" containerID="af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f" Oct 30 20:59:46.249507 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:59:46.249478 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f\": container with ID starting with af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f not found: ID does not exist" containerID="af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f" Oct 30 20:59:46.249569 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.249521 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f"} err="failed to get container status \"af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f\": rpc error: code = NotFound desc = could not find container \"af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f\": container with ID starting with af9ea35ca7e2a59e94bb69764970ca34fea64eb30ddec56cd24e301e318e6e3f not found: ID does not exist" Oct 30 20:59:46.275056 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.275015 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert\") pod \"995e4665-61b8-494c-8cf4-ecaf186a4922\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " Oct 30 20:59:46.275256 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.275088 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal\" (UniqueName: \"kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal\") pod \"995e4665-61b8-494c-8cf4-ecaf186a4922\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " Oct 30 20:59:46.275760 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.275725 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7\" (UniqueName: \"kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7\") pod \"995e4665-61b8-494c-8cf4-ecaf186a4922\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " Oct 30 20:59:46.275855 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.275780 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs\") pod \"995e4665-61b8-494c-8cf4-ecaf186a4922\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " Oct 30 20:59:46.275855 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.275842 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs\") pod \"995e4665-61b8-494c-8cf4-ecaf186a4922\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " Oct 30 20:59:46.275961 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.275886 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs\") pod \"995e4665-61b8-494c-8cf4-ecaf186a4922\" (UID: \"995e4665-61b8-494c-8cf4-ecaf186a4922\") " Oct 30 20:59:46.286451 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.279846 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert" pod "995e4665-61b8-494c-8cf4-ecaf186a4922" (UID: "995e4665-61b8-494c-8cf4-ecaf186a4922"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:46.286451 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.280090 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal" pod "995e4665-61b8-494c-8cf4-ecaf186a4922" (UID: "995e4665-61b8-494c-8cf4-ecaf186a4922"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:46.286451 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.280118 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7" pod "995e4665-61b8-494c-8cf4-ecaf186a4922" (UID: "995e4665-61b8-494c-8cf4-ecaf186a4922"). InnerVolumeSpecName "kube-api-access-npbz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:46.286451 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.286102 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs" pod "995e4665-61b8-494c-8cf4-ecaf186a4922" (UID: "995e4665-61b8-494c-8cf4-ecaf186a4922"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:46.286451 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.286184 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs" pod "995e4665-61b8-494c-8cf4-ecaf186a4922" (UID: "995e4665-61b8-494c-8cf4-ecaf186a4922"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:46.287652 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.287623 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs" pod "995e4665-61b8-494c-8cf4-ecaf186a4922" (UID: "995e4665-61b8-494c-8cf4-ecaf186a4922"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:46.377932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.377896 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.377932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.377930 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/995e4665-61b8-494c-8cf4-ecaf186a4922-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.377932 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.377944 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npbz7\" (UniqueName: \"kubernetes.io/projected/995e4665-61b8-494c-8cf4-ecaf186a4922-kube-api-access-npbz7\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.378202 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.377959 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.378202 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.377974 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.378202 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.377986 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/995e4665-61b8-494c-8cf4-ecaf186a4922-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 20:59:46.660867 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.660824 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:46.666216 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.666180 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:46.741878 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.741840 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:46.742254 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.742242 4988 cpu_manager.go:414] "containerMap: removing container" podUID="995e4665-61b8-494c-8cf4-ecaf186a4922" containerName="otc-container" Oct 30 20:59:46.742297 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.742256 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995e4665-61b8-494c-8cf4-ecaf186a4922" containerName="otc-container" Oct 30 20:59:46.742347 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.742338 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="995e4665-61b8-494c-8cf4-ecaf186a4922" containerName="otc-container" Oct 30 20:59:46.746910 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.746889 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.799536 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.799435 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 20:59:46.799536 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.799478 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 20:59:46.799755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.799607 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 20:59:46.799755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.799634 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 20:59:46.799755 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.799729 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 20:59:46.799881 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.799786 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 20:59:46.806028 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.805991 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:46.882609 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.882563 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.882806 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.882632 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf74z\" (UniqueName: \"kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.882806 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.882715 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.882806 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.882750 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.882806 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.882782 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.882943 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.882872 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.983474 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.983403 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.983665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.983484 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.983665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.983525 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.983665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.983566 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf74z\" (UniqueName: \"kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.983665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.983604 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.983665 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.983636 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.984461 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.984430 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.986118 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.986089 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.986242 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.986142 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.986314 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.986293 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:46.986364 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:46.986348 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:47.045148 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:47.045116 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf74z\" (UniqueName: \"kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z\") pod \"data-science-collector-collector-1\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:47.059716 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:47.059620 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 20:59:47.436404 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:47.436371 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 20:59:47.439833 ip-10-0-141-115 kubenswrapper[4988]: W1030 20:59:47.439800 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88db1dd_80bd_43c7_8f90_d61b3404256c.slice/crio-fbf50d2f32853bec9f1cb83bce124039bb77e421fc3c88678e594fb020da7ec5 WatchSource:0}: Error finding container fbf50d2f32853bec9f1cb83bce124039bb77e421fc3c88678e594fb020da7ec5: Status 404 returned error can't find the container with id fbf50d2f32853bec9f1cb83bce124039bb77e421fc3c88678e594fb020da7ec5 Oct 30 20:59:47.441970 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:47.441937 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:59:47.442033 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:47.441999 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 20:59:48.016109 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:48.016069 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995e4665-61b8-494c-8cf4-ecaf186a4922" path="/var/lib/kubelet/pods/995e4665-61b8-494c-8cf4-ecaf186a4922/volumes" Oct 30 20:59:48.253099 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:48.253058 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b88db1dd-80bd-43c7-8f90-d61b3404256c","Type":"ContainerStarted","Data":"92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21"} Oct 30 20:59:48.253320 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:48.253120 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b88db1dd-80bd-43c7-8f90-d61b3404256c","Type":"ContainerStarted","Data":"fbf50d2f32853bec9f1cb83bce124039bb77e421fc3c88678e594fb020da7ec5"} Oct 30 20:59:48.383295 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:48.383233 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.383218644 podStartE2EDuration="2.383218644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:48.382817793 +0000 UTC m=+1416.490313705" watchObservedRunningTime="2026-10-30 20:59:48.383218644 +0000 UTC m=+1416.490714556" Oct 30 20:59:50.031975 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:50.031934 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:50.287952 ip-10-0-141-115 kubenswrapper[4988]: I1030 20:59:50.287861 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:56.395677 ip-10-0-141-115 kubenswrapper[4988]: E1030 20:59:56.395634 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:00:10.574709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:10.574664 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.157116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:11.157081 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.357635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:11.357600 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.502465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:11.502358 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.646739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:11.646690 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.793487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:11.793383 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.940210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:11.940169 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:17.454180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:17.454137 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:17.454844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:17.454365 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b88db1dd-80bd-43c7-8f90-d61b3404256c" containerName="otc-container" containerID="cri-o://92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21" gracePeriod=30 Oct 30 21:00:17.789318 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:17.789231 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:17.981038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:17.980995 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:18.150729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:18.150686 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:19.806504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.806477 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:19.866269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866156 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs\") pod \"b88db1dd-80bd-43c7-8f90-d61b3404256c\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " Oct 30 21:00:19.866269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866224 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs\") pod \"b88db1dd-80bd-43c7-8f90-d61b3404256c\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " Oct 30 21:00:19.866269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866260 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert\") pod \"b88db1dd-80bd-43c7-8f90-d61b3404256c\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " Oct 30 21:00:19.866632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866315 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal\") pod \"b88db1dd-80bd-43c7-8f90-d61b3404256c\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " Oct 30 21:00:19.866632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866337 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z\" (UniqueName: \"kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z\") pod \"b88db1dd-80bd-43c7-8f90-d61b3404256c\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " Oct 30 21:00:19.866632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866359 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs\") pod \"b88db1dd-80bd-43c7-8f90-d61b3404256c\" (UID: \"b88db1dd-80bd-43c7-8f90-d61b3404256c\") " Oct 30 21:00:19.866840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.866806 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal" pod "b88db1dd-80bd-43c7-8f90-d61b3404256c" (UID: "b88db1dd-80bd-43c7-8f90-d61b3404256c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:19.868764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.868734 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs" pod "b88db1dd-80bd-43c7-8f90-d61b3404256c" (UID: "b88db1dd-80bd-43c7-8f90-d61b3404256c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:19.868891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.868766 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z" pod "b88db1dd-80bd-43c7-8f90-d61b3404256c" (UID: "b88db1dd-80bd-43c7-8f90-d61b3404256c"). InnerVolumeSpecName "kube-api-access-gf74z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:19.868891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.868792 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert" pod "b88db1dd-80bd-43c7-8f90-d61b3404256c" (UID: "b88db1dd-80bd-43c7-8f90-d61b3404256c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:19.868973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.868951 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs" pod "b88db1dd-80bd-43c7-8f90-d61b3404256c" (UID: "b88db1dd-80bd-43c7-8f90-d61b3404256c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:19.869005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.868970 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs" pod "b88db1dd-80bd-43c7-8f90-d61b3404256c" (UID: "b88db1dd-80bd-43c7-8f90-d61b3404256c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:19.967060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.967015 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.967060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.967051 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.967060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.967063 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b88db1dd-80bd-43c7-8f90-d61b3404256c-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.967301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.967074 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf74z\" (UniqueName: \"kubernetes.io/projected/b88db1dd-80bd-43c7-8f90-d61b3404256c-kube-api-access-gf74z\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.967301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.967085 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:19.967301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:19.967093 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b88db1dd-80bd-43c7-8f90-d61b3404256c-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.260964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.260924 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:20.410163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.410118 4988 generic.go:356] "Generic (PLEG): container finished" podID="b88db1dd-80bd-43c7-8f90-d61b3404256c" containerID="92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21" exitCode=0 Oct 30 21:00:20.410341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.410179 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.410341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.410183 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b88db1dd-80bd-43c7-8f90-d61b3404256c","Type":"ContainerDied","Data":"92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21"} Oct 30 21:00:20.410341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.410236 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b88db1dd-80bd-43c7-8f90-d61b3404256c","Type":"ContainerDied","Data":"fbf50d2f32853bec9f1cb83bce124039bb77e421fc3c88678e594fb020da7ec5"} Oct 30 21:00:20.410341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.410260 4988 scope.go:122] "RemoveContainer" containerID="92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21" Oct 30 21:00:20.419473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.419449 4988 scope.go:122] "RemoveContainer" containerID="92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21" Oct 30 21:00:20.419803 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:00:20.419783 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21\": container with ID starting with 92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21 not found: ID does not exist" containerID="92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21" Oct 30 21:00:20.419875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.419818 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21"} err="failed to get container status \"92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21\": rpc error: code = NotFound desc = could not find container \"92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21\": container with ID starting with 92659d7e029b0cf21b875c279263cf6e7cd24f700274bff962704564392b0d21 not found: ID does not exist" Oct 30 21:00:20.553003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.552908 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:20.583003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.582964 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:20.587039 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.587005 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:20.661581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.661542 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:20.662042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.662024 4988 cpu_manager.go:414] "containerMap: removing container" podUID="b88db1dd-80bd-43c7-8f90-d61b3404256c" containerName="otc-container" Oct 30 21:00:20.662135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.662044 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88db1dd-80bd-43c7-8f90-d61b3404256c" containerName="otc-container" Oct 30 21:00:20.662195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.662167 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88db1dd-80bd-43c7-8f90-d61b3404256c" containerName="otc-container" Oct 30 21:00:20.666767 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.666737 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.720819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.720789 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\"" Oct 30 21:00:20.724757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.724721 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:20.766842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.766815 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:00:20.766842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.766831 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:00:20.766842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.766820 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:00:20.767111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.766908 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:00:20.767111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.766932 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:00:20.773942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.773912 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtdt\" (UniqueName: \"kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.774030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.773968 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.774086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.774054 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.774127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.774084 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.774169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.774153 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.774205 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.774183 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.875484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.875365 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.875484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.875434 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.875484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.875454 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.875484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.875489 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.876102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.875514 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.876102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.875671 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtdt\" (UniqueName: \"kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.876304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.876278 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.878061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.878040 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.878176 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.878153 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.878322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.878306 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.878450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.878405 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.937171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.937127 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtdt\" (UniqueName: \"kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt\") pod \"data-science-collector-collector-1\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:20.979492 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:20.979445 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:21.356088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:21.356033 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:21.358784 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:00:21.358755 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d76d612_f1cb_4182_acd9_4678d672b922.slice/crio-c6c70d135082cb42bc21245c1d56c76fee3fca42edc7856eca1146ae00a72aad WatchSource:0}: Error finding container c6c70d135082cb42bc21245c1d56c76fee3fca42edc7856eca1146ae00a72aad: Status 404 returned error can't find the container with id c6c70d135082cb42bc21245c1d56c76fee3fca42edc7856eca1146ae00a72aad Oct 30 21:00:21.360775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:21.360744 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:00:21.360845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:21.360805 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:00:21.416854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:21.416815 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d76d612-f1cb-4182-acd9-4678d672b922","Type":"ContainerStarted","Data":"c6c70d135082cb42bc21245c1d56c76fee3fca42edc7856eca1146ae00a72aad"} Oct 30 21:00:22.015935 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:22.015897 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88db1dd-80bd-43c7-8f90-d61b3404256c" path="/var/lib/kubelet/pods/b88db1dd-80bd-43c7-8f90-d61b3404256c/volumes" Oct 30 21:00:22.424290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:22.424254 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d76d612-f1cb-4182-acd9-4678d672b922","Type":"ContainerStarted","Data":"6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595"} Oct 30 21:00:22.543044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:22.542973 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.542953933 podStartE2EDuration="2.542953933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:22.54133889 +0000 UTC m=+1450.648834815" watchObservedRunningTime="2026-10-30 21:00:22.542953933 +0000 UTC m=+1450.650449847" Oct 30 21:00:26.394304 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:00:26.394266 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:00:44.961753 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:44.961713 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.107452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:45.107395 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.308288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:45.308194 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.455355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:45.455312 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.599097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:45.598993 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.743838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:45.743796 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:50.709299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:50.709254 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:50.709813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:50.709475 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2d76d612-f1cb-4182-acd9-4678d672b922" containerName="otc-container" containerID="cri-o://6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595" gracePeriod=30 Oct 30 21:00:51.040198 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:51.040103 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:51.185213 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:51.185171 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:51.489316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:51.489278 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:53.062298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.062267 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.165905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.165872 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs\") pod \"2d76d612-f1cb-4182-acd9-4678d672b922\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " Oct 30 21:00:53.166102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.165952 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt\" (UniqueName: \"kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt\") pod \"2d76d612-f1cb-4182-acd9-4678d672b922\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " Oct 30 21:00:53.166102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.166056 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert\") pod \"2d76d612-f1cb-4182-acd9-4678d672b922\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " Oct 30 21:00:53.166197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.166114 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal\") pod \"2d76d612-f1cb-4182-acd9-4678d672b922\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " Oct 30 21:00:53.166197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.166144 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs\") pod \"2d76d612-f1cb-4182-acd9-4678d672b922\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " Oct 30 21:00:53.166197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.166187 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs\") pod \"2d76d612-f1cb-4182-acd9-4678d672b922\" (UID: \"2d76d612-f1cb-4182-acd9-4678d672b922\") " Oct 30 21:00:53.166764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.166570 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal" pod "2d76d612-f1cb-4182-acd9-4678d672b922" (UID: "2d76d612-f1cb-4182-acd9-4678d672b922"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:53.168376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.168352 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert" pod "2d76d612-f1cb-4182-acd9-4678d672b922" (UID: "2d76d612-f1cb-4182-acd9-4678d672b922"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:53.168584 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.168564 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs" pod "2d76d612-f1cb-4182-acd9-4678d672b922" (UID: "2d76d612-f1cb-4182-acd9-4678d672b922"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:53.168699 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.168681 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs" pod "2d76d612-f1cb-4182-acd9-4678d672b922" (UID: "2d76d612-f1cb-4182-acd9-4678d672b922"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:53.168751 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.168708 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt" pod "2d76d612-f1cb-4182-acd9-4678d672b922" (UID: "2d76d612-f1cb-4182-acd9-4678d672b922"). InnerVolumeSpecName "kube-api-access-4vtdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:53.168815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.168797 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs" pod "2d76d612-f1cb-4182-acd9-4678d672b922" (UID: "2d76d612-f1cb-4182-acd9-4678d672b922"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:53.267336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.267293 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.267336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.267328 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtdt\" (UniqueName: \"kubernetes.io/projected/2d76d612-f1cb-4182-acd9-4678d672b922-kube-api-access-4vtdt\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.267336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.267339 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.267336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.267350 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d76d612-f1cb-4182-acd9-4678d672b922-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.267651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.267363 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.267651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.267372 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d76d612-f1cb-4182-acd9-4678d672b922-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:00:53.583602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.583504 4988 generic.go:356] "Generic (PLEG): container finished" podID="2d76d612-f1cb-4182-acd9-4678d672b922" containerID="6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595" exitCode=0 Oct 30 21:00:53.583602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.583555 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.583819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.583594 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d76d612-f1cb-4182-acd9-4678d672b922","Type":"ContainerDied","Data":"6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595"} Oct 30 21:00:53.583819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.583656 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d76d612-f1cb-4182-acd9-4678d672b922","Type":"ContainerDied","Data":"c6c70d135082cb42bc21245c1d56c76fee3fca42edc7856eca1146ae00a72aad"} Oct 30 21:00:53.583819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.583686 4988 scope.go:122] "RemoveContainer" containerID="6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595" Oct 30 21:00:53.593316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.593289 4988 scope.go:122] "RemoveContainer" containerID="6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595" Oct 30 21:00:53.593673 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:00:53.593653 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595\": container with ID starting with 6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595 not found: ID does not exist" containerID="6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595" Oct 30 21:00:53.593738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.593685 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595"} err="failed to get container status \"6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595\": rpc error: code = NotFound desc = could not find container \"6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595\": container with ID starting with 6ab48a3c502edacd31154dbc37a2adbbe3edb8f28ac458190eede6e644583595 not found: ID does not exist" Oct 30 21:00:53.668647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.668615 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:53.747631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.747595 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:53.751654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.751615 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:53.828066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.828024 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:53.828480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.828464 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2d76d612-f1cb-4182-acd9-4678d672b922" containerName="otc-container" Oct 30 21:00:53.828533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.828483 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d76d612-f1cb-4182-acd9-4678d672b922" containerName="otc-container" Oct 30 21:00:53.828587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.828576 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d76d612-f1cb-4182-acd9-4678d672b922" containerName="otc-container" Oct 30 21:00:53.833107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.833083 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.886529 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.886495 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:00:53.887196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.887165 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:00:53.887325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.887265 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:00:53.887325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.887290 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:00:53.887404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.887340 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:00:53.889592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.889576 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:00:53.895458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.895423 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:53.972385 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.972340 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97pr\" (UniqueName: \"kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.972618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.972500 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.972696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.972618 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.972696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.972662 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.972807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.972760 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:53.972807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:53.972789 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.016561 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.016524 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d76d612-f1cb-4182-acd9-4678d672b922" path="/var/lib/kubelet/pods/2d76d612-f1cb-4182-acd9-4678d672b922/volumes" Oct 30 21:00:54.074265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.074221 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.074265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.074265 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.074785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.074301 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.074785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.074445 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.074785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.074509 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97pr\" (UniqueName: \"kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.074785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.074574 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.075146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.075073 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.076921 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.076882 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.077071 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.077048 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.077361 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.077343 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.077399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.077373 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.136269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.136218 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97pr\" (UniqueName: \"kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr\") pod \"data-science-collector-collector-1\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.146988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.146946 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:00:54.524203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.524176 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:00:54.526897 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:00:54.526864 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20ac8d9_8d61_450f_a970_f795ebefa985.slice/crio-2f9fc580d98fbc83b4bc83fd389d07720c4c5ef0294aab89392ff8f177977f25 WatchSource:0}: Error finding container 2f9fc580d98fbc83b4bc83fd389d07720c4c5ef0294aab89392ff8f177977f25: Status 404 returned error can't find the container with id 2f9fc580d98fbc83b4bc83fd389d07720c4c5ef0294aab89392ff8f177977f25 Oct 30 21:00:54.529081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.529042 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:00:54.529178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.529109 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:00:54.590506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.590464 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d20ac8d9-8d61-450f-a970-f795ebefa985","Type":"ContainerStarted","Data":"1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c"} Oct 30 21:00:54.590666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.590535 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d20ac8d9-8d61-450f-a970-f795ebefa985","Type":"ContainerStarted","Data":"2f9fc580d98fbc83b4bc83fd389d07720c4c5ef0294aab89392ff8f177977f25"} Oct 30 21:00:54.708692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:00:54.708630 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.708616313 podStartE2EDuration="1.708616313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:54.707140745 +0000 UTC m=+1482.814636658" watchObservedRunningTime="2026-10-30 21:00:54.708616313 +0000 UTC m=+1482.816112224" Oct 30 21:00:56.395104 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:00:56.394998 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:01:09.880874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:09.880835 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:12.155144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:12.155117 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:01:12.168590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:12.168553 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:01:12.708766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:12.708729 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:12.912723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:12.912686 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:16.883970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:16.883926 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:17.255780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:17.255683 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:17.472521 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:17.472481 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:17.618756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:17.618669 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:17.770506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:17.770463 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:21.576698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:21.576659 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:22.662040 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:22.661997 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:22.662522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:22.662224 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d20ac8d9-8d61-450f-a970-f795ebefa985" containerName="otc-container" containerID="cri-o://1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c" gracePeriod=30 Oct 30 21:01:22.997269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:22.997176 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:23.140230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:23.140189 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:23.353716 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:23.353623 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:25.017202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.017172 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:25.120519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.120389 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr\" (UniqueName: \"kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr\") pod \"d20ac8d9-8d61-450f-a970-f795ebefa985\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " Oct 30 21:01:25.120519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.120510 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs\") pod \"d20ac8d9-8d61-450f-a970-f795ebefa985\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " Oct 30 21:01:25.120828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.120548 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs\") pod \"d20ac8d9-8d61-450f-a970-f795ebefa985\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " Oct 30 21:01:25.120828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.120572 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs\") pod \"d20ac8d9-8d61-450f-a970-f795ebefa985\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " Oct 30 21:01:25.120828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.120606 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert\") pod \"d20ac8d9-8d61-450f-a970-f795ebefa985\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " Oct 30 21:01:25.120828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.120649 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal\") pod \"d20ac8d9-8d61-450f-a970-f795ebefa985\" (UID: \"d20ac8d9-8d61-450f-a970-f795ebefa985\") " Oct 30 21:01:25.121130 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.121096 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal" pod "d20ac8d9-8d61-450f-a970-f795ebefa985" (UID: "d20ac8d9-8d61-450f-a970-f795ebefa985"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:25.122811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.122785 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs" pod "d20ac8d9-8d61-450f-a970-f795ebefa985" (UID: "d20ac8d9-8d61-450f-a970-f795ebefa985"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:25.122811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.122796 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs" pod "d20ac8d9-8d61-450f-a970-f795ebefa985" (UID: "d20ac8d9-8d61-450f-a970-f795ebefa985"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:25.122998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.122843 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr" pod "d20ac8d9-8d61-450f-a970-f795ebefa985" (UID: "d20ac8d9-8d61-450f-a970-f795ebefa985"). InnerVolumeSpecName "kube-api-access-t97pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:25.123121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.123104 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert" pod "d20ac8d9-8d61-450f-a970-f795ebefa985" (UID: "d20ac8d9-8d61-450f-a970-f795ebefa985"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:25.123166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.123143 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs" pod "d20ac8d9-8d61-450f-a970-f795ebefa985" (UID: "d20ac8d9-8d61-450f-a970-f795ebefa985"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:25.222054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.222003 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d20ac8d9-8d61-450f-a970-f795ebefa985-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.222054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.222046 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97pr\" (UniqueName: \"kubernetes.io/projected/d20ac8d9-8d61-450f-a970-f795ebefa985-kube-api-access-t97pr\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.222054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.222058 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.222054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.222067 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.222340 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.222077 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.222340 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.222086 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d20ac8d9-8d61-450f-a970-f795ebefa985-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.743048 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.743006 4988 generic.go:356] "Generic (PLEG): container finished" podID="d20ac8d9-8d61-450f-a970-f795ebefa985" containerID="1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c" exitCode=0 Oct 30 21:01:25.743230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.743052 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:25.743230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.743096 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d20ac8d9-8d61-450f-a970-f795ebefa985","Type":"ContainerDied","Data":"1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c"} Oct 30 21:01:25.743230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.743150 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d20ac8d9-8d61-450f-a970-f795ebefa985","Type":"ContainerDied","Data":"2f9fc580d98fbc83b4bc83fd389d07720c4c5ef0294aab89392ff8f177977f25"} Oct 30 21:01:25.743230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.743173 4988 scope.go:122] "RemoveContainer" containerID="1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c" Oct 30 21:01:25.752555 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.752530 4988 scope.go:122] "RemoveContainer" containerID="1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c" Oct 30 21:01:25.752919 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:01:25.752894 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c\": container with ID starting with 1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c not found: ID does not exist" containerID="1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c" Oct 30 21:01:25.752970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.752931 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c"} err="failed to get container status \"1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c\": rpc error: code = NotFound desc = could not find container \"1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c\": container with ID starting with 1c1446d74649e7039ee536ba48a17b1e0dcf674eb15964eadcc171eb39ed151c not found: ID does not exist" Oct 30 21:01:25.907519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.907478 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:25.957168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.957129 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:25.991967 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.991929 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:25.992604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.992587 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d20ac8d9-8d61-450f-a970-f795ebefa985" containerName="otc-container" Oct 30 21:01:25.992659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.992606 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20ac8d9-8d61-450f-a970-f795ebefa985" containerName="otc-container" Oct 30 21:01:25.992698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.992676 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20ac8d9-8d61-450f-a970-f795ebefa985" containerName="otc-container" Oct 30 21:01:25.997151 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:25.997099 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.015988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.015949 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20ac8d9-8d61-450f-a970-f795ebefa985" path="/var/lib/kubelet/pods/d20ac8d9-8d61-450f-a970-f795ebefa985/volumes" Oct 30 21:01:26.027746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.027707 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.027746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.027753 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.028227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.027782 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbxl\" (UniqueName: \"kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.028227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.027938 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.028227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.027982 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.028227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.028010 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.050100 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.050064 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:01:26.050282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.050119 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:01:26.050282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.050136 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:01:26.050282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.050172 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:01:26.050282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.050227 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:01:26.050282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.050270 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:01:26.055328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.055298 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:26.128972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.128932 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.128972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.128977 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.129274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.129010 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbxl\" (UniqueName: \"kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.129274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.129042 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.129274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.129092 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.129274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.129122 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.129989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.129957 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.131708 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.131683 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.131836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.131749 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.131836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.131751 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.131913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.131854 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.190131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.190094 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbxl\" (UniqueName: \"kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl\") pod \"data-science-collector-collector-1\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.310687 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.310582 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:26.397727 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:01:26.397670 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:01:26.691334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.691299 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:26.694565 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:01:26.694530 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1eacc3_3bb7_49ad_8e30_57335a4a4b2e.slice/crio-6ef234ad866709d1f361b6095c09cf4cb9fb44d99c16f2f3470d8e0fa35d7684 WatchSource:0}: Error finding container 6ef234ad866709d1f361b6095c09cf4cb9fb44d99c16f2f3470d8e0fa35d7684: Status 404 returned error can't find the container with id 6ef234ad866709d1f361b6095c09cf4cb9fb44d99c16f2f3470d8e0fa35d7684 Oct 30 21:01:26.696918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.696881 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:01:26.697048 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.696961 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:01:26.750863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:26.750834 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e","Type":"ContainerStarted","Data":"6ef234ad866709d1f361b6095c09cf4cb9fb44d99c16f2f3470d8e0fa35d7684"} Oct 30 21:01:27.757203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:27.757169 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e","Type":"ContainerStarted","Data":"f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176"} Oct 30 21:01:27.876577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:27.876523 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.876504898 podStartE2EDuration="2.876504898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:27.875125438 +0000 UTC m=+1515.982621352" watchObservedRunningTime="2026-10-30 21:01:27.876504898 +0000 UTC m=+1515.984000877" Oct 30 21:01:47.971710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:47.971669 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.174638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:48.174595 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.326837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:48.326742 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.481554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:48.481516 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.647374 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:48.647335 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.803180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:48.803145 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:53.714089 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:53.714039 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:53.714556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:53.714302 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" containerName="otc-container" containerID="cri-o://f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176" gracePeriod=30 Oct 30 21:01:54.048025 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:54.047942 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:54.190559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:54.190518 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:54.402516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:54.402478 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:56.058047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.058020 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:56.210137 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210041 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs\") pod \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " Oct 30 21:01:56.210137 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210096 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs\") pod \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " Oct 30 21:01:56.210387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210141 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal\") pod \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " Oct 30 21:01:56.210387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210188 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl\" (UniqueName: \"kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl\") pod \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " Oct 30 21:01:56.210387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210263 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert\") pod \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " Oct 30 21:01:56.210387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210330 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs\") pod \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\" (UID: \"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e\") " Oct 30 21:01:56.210651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.210619 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal" pod "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" (UID: "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:56.212466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.212391 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert" pod "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" (UID: "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:56.212591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.212510 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs" pod "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" (UID: "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:56.212591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.212535 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs" pod "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" (UID: "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:56.212794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.212777 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs" pod "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" (UID: "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:56.212870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.212843 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl" pod "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" (UID: "ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e"). InnerVolumeSpecName "kube-api-access-fdbxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:56.311556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.311501 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbxl\" (UniqueName: \"kubernetes.io/projected/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-kube-api-access-fdbxl\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:56.311556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.311553 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:56.311556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.311578 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:56.311827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.311592 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:56.311827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.311605 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:56.311827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.311617 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:01:56.395133 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:01:56.395091 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:01:56.897866 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.897827 4988 generic.go:356] "Generic (PLEG): container finished" podID="ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" containerID="f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176" exitCode=0 Oct 30 21:01:56.898052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.897887 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:56.898052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.897909 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e","Type":"ContainerDied","Data":"f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176"} Oct 30 21:01:56.898052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.897993 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e","Type":"ContainerDied","Data":"6ef234ad866709d1f361b6095c09cf4cb9fb44d99c16f2f3470d8e0fa35d7684"} Oct 30 21:01:56.898052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.898028 4988 scope.go:122] "RemoveContainer" containerID="f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176" Oct 30 21:01:56.907460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.907430 4988 scope.go:122] "RemoveContainer" containerID="f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176" Oct 30 21:01:56.907873 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:01:56.907845 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176\": container with ID starting with f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176 not found: ID does not exist" containerID="f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176" Oct 30 21:01:56.907951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:56.907889 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176"} err="failed to get container status \"f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176\": rpc error: code = NotFound desc = could not find container \"f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176\": container with ID starting with f9ac1cc6258d8f0ff74643aa2a4a82c1ce24380b28d4cabde76240f749ed0176 not found: ID does not exist" Oct 30 21:01:57.076639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.076600 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:57.126119 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.126079 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:57.161487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.161387 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:57.161880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.161866 4988 cpu_manager.go:414] "containerMap: removing container" podUID="ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" containerName="otc-container" Oct 30 21:01:57.161926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.161881 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" containerName="otc-container" Oct 30 21:01:57.162002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.161992 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" containerName="otc-container" Oct 30 21:01:57.166574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.166540 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.222873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.222253 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:01:57.223067 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.222907 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:01:57.271779 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.271745 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:01:57.271978 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.271780 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:01:57.271978 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.271862 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:01:57.271978 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.271883 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:01:57.273371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.273350 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:57.320712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.320670 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.320712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.320712 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.320958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.320740 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.320958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.320812 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.320958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.320888 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbk7\" (UniqueName: \"kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.320958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.320939 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.421665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.421556 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.421665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.421640 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.421665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.421669 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.421953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.421705 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.421953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.421733 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.421953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.421797 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbk7\" (UniqueName: \"kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.422565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.422534 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.424289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.424259 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.424426 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.424389 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.424518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.424497 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.424588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.424573 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.482833 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.482792 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbk7\" (UniqueName: \"kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7\") pod \"data-science-collector-collector-1\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:57.779864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:57.779766 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:01:58.017067 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:58.017031 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e" path="/var/lib/kubelet/pods/ab1eacc3-3bb7-49ad-8e30-57335a4a4b2e/volumes" Oct 30 21:01:58.158457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:58.158405 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:01:58.161649 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:01:58.161618 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493eb103_4566_4d62_8aa8_1008282054fd.slice/crio-3fd353d939550caffcfaf20b511f136fb699c2c9dc70621c5f15aab7540620a5 WatchSource:0}: Error finding container 3fd353d939550caffcfaf20b511f136fb699c2c9dc70621c5f15aab7540620a5: Status 404 returned error can't find the container with id 3fd353d939550caffcfaf20b511f136fb699c2c9dc70621c5f15aab7540620a5 Oct 30 21:01:58.163808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:58.163773 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:01:58.163895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:58.163839 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:01:58.911338 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:58.911299 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"493eb103-4566-4d62-8aa8-1008282054fd","Type":"ContainerStarted","Data":"1e6db588dbdb05d587523a3a7e5a7ec6c50b87dda87c12ab8f1116486913a0b2"} Oct 30 21:01:58.911556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:58.911358 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"493eb103-4566-4d62-8aa8-1008282054fd","Type":"ContainerStarted","Data":"3fd353d939550caffcfaf20b511f136fb699c2c9dc70621c5f15aab7540620a5"} Oct 30 21:01:59.032098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:01:59.032036 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.032021442 podStartE2EDuration="2.032021442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:59.030529989 +0000 UTC m=+1547.138025901" watchObservedRunningTime="2026-10-30 21:01:59.032021442 +0000 UTC m=+1547.139517353" Oct 30 21:02:10.382824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:10.382775 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:10.564509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:10.564476 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:13.937349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:13.937297 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:19.342611 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:19.342573 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:19.493748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:19.493699 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:19.694920 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:19.694865 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:19.895925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:19.895888 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:20.040531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:20.040436 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:20.189576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:20.189540 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:25.336838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:25.336758 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:25.550213 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:25.550174 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:25.694660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:25.694621 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:26.394739 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:02:26.394700 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:02:31.843761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:31.843723 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:31.844166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:31.843951 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="493eb103-4566-4d62-8aa8-1008282054fd" containerName="otc-container" containerID="cri-o://1e6db588dbdb05d587523a3a7e5a7ec6c50b87dda87c12ab8f1116486913a0b2" gracePeriod=30 Oct 30 21:02:34.080249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.080202 4988 generic.go:356] "Generic (PLEG): container finished" podID="493eb103-4566-4d62-8aa8-1008282054fd" containerID="1e6db588dbdb05d587523a3a7e5a7ec6c50b87dda87c12ab8f1116486913a0b2" exitCode=0 Oct 30 21:02:34.080637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.080280 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"493eb103-4566-4d62-8aa8-1008282054fd","Type":"ContainerDied","Data":"1e6db588dbdb05d587523a3a7e5a7ec6c50b87dda87c12ab8f1116486913a0b2"} Oct 30 21:02:34.199958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.199926 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:34.320290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320194 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal\") pod \"493eb103-4566-4d62-8aa8-1008282054fd\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " Oct 30 21:02:34.320290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320266 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs\") pod \"493eb103-4566-4d62-8aa8-1008282054fd\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " Oct 30 21:02:34.320290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320292 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7\" (UniqueName: \"kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7\") pod \"493eb103-4566-4d62-8aa8-1008282054fd\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " Oct 30 21:02:34.320638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320313 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs\") pod \"493eb103-4566-4d62-8aa8-1008282054fd\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " Oct 30 21:02:34.320638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320489 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs\") pod \"493eb103-4566-4d62-8aa8-1008282054fd\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " Oct 30 21:02:34.320752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320708 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal" pod "493eb103-4566-4d62-8aa8-1008282054fd" (UID: "493eb103-4566-4d62-8aa8-1008282054fd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:34.320902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.320550 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert\") pod \"493eb103-4566-4d62-8aa8-1008282054fd\" (UID: \"493eb103-4566-4d62-8aa8-1008282054fd\") " Oct 30 21:02:34.321244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.321211 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/493eb103-4566-4d62-8aa8-1008282054fd-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:02:34.324575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.324534 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs" pod "493eb103-4566-4d62-8aa8-1008282054fd" (UID: "493eb103-4566-4d62-8aa8-1008282054fd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:34.324575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.324561 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7" pod "493eb103-4566-4d62-8aa8-1008282054fd" (UID: "493eb103-4566-4d62-8aa8-1008282054fd"). InnerVolumeSpecName "kube-api-access-pbbk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:34.324727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.324566 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs" pod "493eb103-4566-4d62-8aa8-1008282054fd" (UID: "493eb103-4566-4d62-8aa8-1008282054fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:34.324792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.324768 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert" pod "493eb103-4566-4d62-8aa8-1008282054fd" (UID: "493eb103-4566-4d62-8aa8-1008282054fd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:34.333600 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.333560 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs" pod "493eb103-4566-4d62-8aa8-1008282054fd" (UID: "493eb103-4566-4d62-8aa8-1008282054fd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:34.422637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.422597 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:02:34.422637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.422629 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbbk7\" (UniqueName: \"kubernetes.io/projected/493eb103-4566-4d62-8aa8-1008282054fd-kube-api-access-pbbk7\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:02:34.422637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.422639 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:02:34.422637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.422647 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:02:34.422907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:34.422656 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/493eb103-4566-4d62-8aa8-1008282054fd-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:02:35.086452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.086422 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.086452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.086440 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"493eb103-4566-4d62-8aa8-1008282054fd","Type":"ContainerDied","Data":"3fd353d939550caffcfaf20b511f136fb699c2c9dc70621c5f15aab7540620a5"} Oct 30 21:02:35.086948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.086505 4988 scope.go:122] "RemoveContainer" containerID="1e6db588dbdb05d587523a3a7e5a7ec6c50b87dda87c12ab8f1116486913a0b2" Oct 30 21:02:35.275987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.275947 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:35.325846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.325804 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:35.380742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.380663 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:35.381071 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.381058 4988 cpu_manager.go:414] "containerMap: removing container" podUID="493eb103-4566-4d62-8aa8-1008282054fd" containerName="otc-container" Oct 30 21:02:35.381112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.381072 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493eb103-4566-4d62-8aa8-1008282054fd" containerName="otc-container" Oct 30 21:02:35.381152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.381144 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="493eb103-4566-4d62-8aa8-1008282054fd" containerName="otc-container" Oct 30 21:02:35.385707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.385687 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.431215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.431172 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.431435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.431249 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.431435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.431378 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.431564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.431446 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp6sp\" (UniqueName: \"kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.431564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.431522 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.431656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.431561 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.438347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.438301 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:02:35.438347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.438308 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:02:35.438644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.438498 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:02:35.438644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.438535 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:02:35.438644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.438620 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:02:35.441809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.441782 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:02:35.489037 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.489002 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:35.532553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.532515 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.532756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.532577 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.532756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.532609 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.532756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.532624 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp6sp\" (UniqueName: \"kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.532756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.532647 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.532756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.532661 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.533608 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.533577 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.535213 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.535189 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.535337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.535319 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.535382 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.535349 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.535443 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.535400 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.593957 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.593926 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp6sp\" (UniqueName: \"kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp\") pod \"data-science-collector-collector-1\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:35.698250 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:35.698208 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:02:36.017701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:36.017613 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493eb103-4566-4d62-8aa8-1008282054fd" path="/var/lib/kubelet/pods/493eb103-4566-4d62-8aa8-1008282054fd/volumes" Oct 30 21:02:36.074603 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:36.074566 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:36.078454 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:02:36.078426 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb893f20_4efe_41ef_b760_5bd641f8cd37.slice/crio-f238050afa33dbeabfe35ad117c0b03f1bc19ff35b293c4945278f8db5e11da3 WatchSource:0}: Error finding container f238050afa33dbeabfe35ad117c0b03f1bc19ff35b293c4945278f8db5e11da3: Status 404 returned error can't find the container with id f238050afa33dbeabfe35ad117c0b03f1bc19ff35b293c4945278f8db5e11da3 Oct 30 21:02:36.080640 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:36.080607 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:02:36.080722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:36.080669 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:02:36.094685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:36.094654 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db893f20-4efe-41ef-b760-5bd641f8cd37","Type":"ContainerStarted","Data":"f238050afa33dbeabfe35ad117c0b03f1bc19ff35b293c4945278f8db5e11da3"} Oct 30 21:02:37.100728 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:37.100688 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db893f20-4efe-41ef-b760-5bd641f8cd37","Type":"ContainerStarted","Data":"ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44"} Oct 30 21:02:37.222620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:37.222560 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.222547444 podStartE2EDuration="2.222547444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:37.220654112 +0000 UTC m=+1585.328150053" watchObservedRunningTime="2026-10-30 21:02:37.222547444 +0000 UTC m=+1585.330043356" Oct 30 21:02:52.516885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:52.516849 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:52.707710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:52.707676 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:52.852457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:52.852341 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:53.068457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:53.068397 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:53.228210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:53.228169 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:53.371588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:53.371551 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:56.395502 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:02:56.395466 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:02:58.308884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:58.308842 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:02:58.309345 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:58.309061 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="db893f20-4efe-41ef-b760-5bd641f8cd37" containerName="otc-container" containerID="cri-o://ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44" gracePeriod=30 Oct 30 21:02:58.639766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:58.639730 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:58.839577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:58.839533 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:58.998858 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:02:58.998757 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:00.659570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.659530 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:00.775574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.775308 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal\" (UniqueName: \"kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal\") pod \"db893f20-4efe-41ef-b760-5bd641f8cd37\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " Oct 30 21:03:00.775574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.775386 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs\") pod \"db893f20-4efe-41ef-b760-5bd641f8cd37\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " Oct 30 21:03:00.775574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.775436 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert\") pod \"db893f20-4efe-41ef-b760-5bd641f8cd37\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " Oct 30 21:03:00.775574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.775477 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs\") pod \"db893f20-4efe-41ef-b760-5bd641f8cd37\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " Oct 30 21:03:00.775971 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.775790 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs\") pod \"db893f20-4efe-41ef-b760-5bd641f8cd37\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " Oct 30 21:03:00.775971 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.775881 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp\" (UniqueName: \"kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp\") pod \"db893f20-4efe-41ef-b760-5bd641f8cd37\" (UID: \"db893f20-4efe-41ef-b760-5bd641f8cd37\") " Oct 30 21:03:00.776312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.776259 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal" pod "db893f20-4efe-41ef-b760-5bd641f8cd37" (UID: "db893f20-4efe-41ef-b760-5bd641f8cd37"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:00.778544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.778513 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert" pod "db893f20-4efe-41ef-b760-5bd641f8cd37" (UID: "db893f20-4efe-41ef-b760-5bd641f8cd37"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:00.778675 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.778609 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs" pod "db893f20-4efe-41ef-b760-5bd641f8cd37" (UID: "db893f20-4efe-41ef-b760-5bd641f8cd37"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:00.778741 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.778697 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs" pod "db893f20-4efe-41ef-b760-5bd641f8cd37" (UID: "db893f20-4efe-41ef-b760-5bd641f8cd37"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:00.779016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.778995 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs" pod "db893f20-4efe-41ef-b760-5bd641f8cd37" (UID: "db893f20-4efe-41ef-b760-5bd641f8cd37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:00.779631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.779605 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp" pod "db893f20-4efe-41ef-b760-5bd641f8cd37" (UID: "db893f20-4efe-41ef-b760-5bd641f8cd37"). InnerVolumeSpecName "kube-api-access-cp6sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:00.877243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.877195 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.877243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.877235 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp6sp\" (UniqueName: \"kubernetes.io/projected/db893f20-4efe-41ef-b760-5bd641f8cd37-kube-api-access-cp6sp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.877243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.877249 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db893f20-4efe-41ef-b760-5bd641f8cd37-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.877533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.877259 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.877533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.877270 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.877533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:00.877279 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db893f20-4efe-41ef-b760-5bd641f8cd37-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.219986 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.219951 4988 generic.go:356] "Generic (PLEG): container finished" podID="db893f20-4efe-41ef-b760-5bd641f8cd37" containerID="ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44" exitCode=0 Oct 30 21:03:01.220199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.220026 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.220199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.220039 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db893f20-4efe-41ef-b760-5bd641f8cd37","Type":"ContainerDied","Data":"ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44"} Oct 30 21:03:01.220199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.220092 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"db893f20-4efe-41ef-b760-5bd641f8cd37","Type":"ContainerDied","Data":"f238050afa33dbeabfe35ad117c0b03f1bc19ff35b293c4945278f8db5e11da3"} Oct 30 21:03:01.220199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.220114 4988 scope.go:122] "RemoveContainer" containerID="ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44" Oct 30 21:03:01.229541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.229517 4988 scope.go:122] "RemoveContainer" containerID="ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44" Oct 30 21:03:01.229889 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:03:01.229860 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44\": container with ID starting with ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44 not found: ID does not exist" containerID="ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44" Oct 30 21:03:01.229970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.229893 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44"} err="failed to get container status \"ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44\": rpc error: code = NotFound desc = could not find container \"ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44\": container with ID starting with ec837c489f84554ca575b638c134cf15215f55eefcc64c69f49836f3314abe44 not found: ID does not exist" Oct 30 21:03:01.384446 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.384395 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:01.390255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.390227 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:01.466628 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.466593 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:01.466990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.466977 4988 cpu_manager.go:414] "containerMap: removing container" podUID="db893f20-4efe-41ef-b760-5bd641f8cd37" containerName="otc-container" Oct 30 21:03:01.467037 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.466990 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db893f20-4efe-41ef-b760-5bd641f8cd37" containerName="otc-container" Oct 30 21:03:01.467075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.467066 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="db893f20-4efe-41ef-b760-5bd641f8cd37" containerName="otc-container" Oct 30 21:03:01.470164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.470121 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.523008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.522975 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:03:01.523280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.523254 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:03:01.523432 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.523273 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:03:01.523432 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.523288 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:03:01.523432 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.523297 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:03:01.523432 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.523302 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:03:01.531841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.531811 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:01.583063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.583024 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztn2n\" (UniqueName: \"kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.583269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.583082 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.583269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.583102 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.583269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.583216 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.583403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.583297 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.583403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.583333 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.684464 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.684393 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.684923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.684489 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztn2n\" (UniqueName: \"kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.684923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.684523 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.684923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.684541 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.684923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.684559 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.684923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.684590 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.685359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.685335 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.687002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.686974 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.687140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.687118 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.687379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.687363 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.687444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.687372 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.745846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.745762 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztn2n\" (UniqueName: \"kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n\") pod \"data-science-collector-collector-1\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:01.783482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:01.783447 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:02.016096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:02.015999 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db893f20-4efe-41ef-b760-5bd641f8cd37" path="/var/lib/kubelet/pods/db893f20-4efe-41ef-b760-5bd641f8cd37/volumes" Oct 30 21:03:02.160473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:02.160438 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:02.164071 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:03:02.164044 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3b3199_3ad6_444e_9fe4_9be055b5a170.slice/crio-544eba339613194596711fcb21ee1300866ce01013226c8baccea7d998c3db09 WatchSource:0}: Error finding container 544eba339613194596711fcb21ee1300866ce01013226c8baccea7d998c3db09: Status 404 returned error can't find the container with id 544eba339613194596711fcb21ee1300866ce01013226c8baccea7d998c3db09 Oct 30 21:03:02.166255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:02.166221 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:03:02.166315 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:02.166289 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:03:02.227124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:02.227087 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f3b3199-3ad6-444e-9fe4-9be055b5a170","Type":"ContainerStarted","Data":"544eba339613194596711fcb21ee1300866ce01013226c8baccea7d998c3db09"} Oct 30 21:03:02.955106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:02.955068 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:03.199044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:03.199013 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:03.234734 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:03.234636 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f3b3199-3ad6-444e-9fe4-9be055b5a170","Type":"ContainerStarted","Data":"fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d"} Oct 30 21:03:03.366174 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:03.366113 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.366099505 podStartE2EDuration="2.366099505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:03.364866256 +0000 UTC m=+1611.472362181" watchObservedRunningTime="2026-10-30 21:03:03.366099505 +0000 UTC m=+1611.473595417" Oct 30 21:03:05.351531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:05.351497 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:08.700941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:08.700891 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:08.904565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:08.904526 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:10.566115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:10.566075 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:11.564571 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:11.564527 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:11.779294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:11.779255 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:18.532317 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:18.532277 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:18.676224 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:18.676182 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:24.945959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:24.945895 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:25.093304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:25.093264 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:26.395659 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:03:26.395621 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:03:35.659445 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:35.659385 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:35.858853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:35.858817 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:36.007299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:36.007213 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:36.225337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:36.225294 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:36.375480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:36.375385 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:36.526941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:36.526902 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:41.386504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:41.386466 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:41.386941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:41.386660 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1f3b3199-3ad6-444e-9fe4-9be055b5a170" containerName="otc-container" containerID="cri-o://fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d" gracePeriod=30 Oct 30 21:03:41.721617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:41.721529 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:41.928973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:41.928928 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:42.108356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:42.108271 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:43.744783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.744753 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:43.854259 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854163 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs\") pod \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " Oct 30 21:03:43.854259 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854231 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs\") pod \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " Oct 30 21:03:43.854516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854261 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs\") pod \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " Oct 30 21:03:43.854516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854293 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert\") pod \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " Oct 30 21:03:43.854516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854340 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n\" (UniqueName: \"kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n\") pod \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " Oct 30 21:03:43.854516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854383 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal\") pod \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\" (UID: \"1f3b3199-3ad6-444e-9fe4-9be055b5a170\") " Oct 30 21:03:43.854943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.854902 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal" pod "1f3b3199-3ad6-444e-9fe4-9be055b5a170" (UID: "1f3b3199-3ad6-444e-9fe4-9be055b5a170"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:43.856722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.856684 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert" pod "1f3b3199-3ad6-444e-9fe4-9be055b5a170" (UID: "1f3b3199-3ad6-444e-9fe4-9be055b5a170"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:43.856845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.856739 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs" pod "1f3b3199-3ad6-444e-9fe4-9be055b5a170" (UID: "1f3b3199-3ad6-444e-9fe4-9be055b5a170"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:43.856845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.856732 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n" pod "1f3b3199-3ad6-444e-9fe4-9be055b5a170" (UID: "1f3b3199-3ad6-444e-9fe4-9be055b5a170"). InnerVolumeSpecName "kube-api-access-ztn2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:43.857045 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.857022 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs" pod "1f3b3199-3ad6-444e-9fe4-9be055b5a170" (UID: "1f3b3199-3ad6-444e-9fe4-9be055b5a170"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:43.857690 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.857668 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs" pod "1f3b3199-3ad6-444e-9fe4-9be055b5a170" (UID: "1f3b3199-3ad6-444e-9fe4-9be055b5a170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:43.955277 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.955237 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:43.955277 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.955272 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:43.955277 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.955285 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:43.955559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.955294 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f3b3199-3ad6-444e-9fe4-9be055b5a170-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:43.955559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.955304 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztn2n\" (UniqueName: \"kubernetes.io/projected/1f3b3199-3ad6-444e-9fe4-9be055b5a170-kube-api-access-ztn2n\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:43.955559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:43.955314 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f3b3199-3ad6-444e-9fe4-9be055b5a170-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.421655 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.421617 4988 generic.go:356] "Generic (PLEG): container finished" podID="1f3b3199-3ad6-444e-9fe4-9be055b5a170" containerID="fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d" exitCode=0 Oct 30 21:03:44.421837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.421665 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.421837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.421706 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f3b3199-3ad6-444e-9fe4-9be055b5a170","Type":"ContainerDied","Data":"fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d"} Oct 30 21:03:44.421837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.421753 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f3b3199-3ad6-444e-9fe4-9be055b5a170","Type":"ContainerDied","Data":"544eba339613194596711fcb21ee1300866ce01013226c8baccea7d998c3db09"} Oct 30 21:03:44.421837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.421775 4988 scope.go:122] "RemoveContainer" containerID="fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d" Oct 30 21:03:44.430927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.430903 4988 scope.go:122] "RemoveContainer" containerID="fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d" Oct 30 21:03:44.431229 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:03:44.431210 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d\": container with ID starting with fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d not found: ID does not exist" containerID="fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d" Oct 30 21:03:44.431279 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.431239 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d"} err="failed to get container status \"fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d\": rpc error: code = NotFound desc = could not find container \"fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d\": container with ID starting with fcb1acfd56eca027d94300b8ea824814a64118e8cc73547fc8b2d18362b4e45d not found: ID does not exist" Oct 30 21:03:44.594062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.594024 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:44.642818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.642779 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:44.688853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.688771 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:44.689170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.689156 4988 cpu_manager.go:414] "containerMap: removing container" podUID="1f3b3199-3ad6-444e-9fe4-9be055b5a170" containerName="otc-container" Oct 30 21:03:44.689212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.689171 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3b3199-3ad6-444e-9fe4-9be055b5a170" containerName="otc-container" Oct 30 21:03:44.689264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.689255 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3b3199-3ad6-444e-9fe4-9be055b5a170" containerName="otc-container" Oct 30 21:03:44.693798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.693768 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.747710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.747669 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:03:44.748157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.747723 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:03:44.748157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.747767 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:03:44.748157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.747846 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:03:44.748157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.747856 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:03:44.748157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.747892 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:03:44.755726 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.755682 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:44.861832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.861790 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.862022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.861860 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.862022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.861918 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.862022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.861996 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djhb\" (UniqueName: \"kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.862144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.862050 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.862144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.862084 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.963233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.963128 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.963233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.963184 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.963233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.963219 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djhb\" (UniqueName: \"kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.963558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.963257 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.963558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.963295 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.963558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.963330 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.964110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.964082 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.965746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.965724 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.965907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.965884 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.965951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.965890 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:44.966126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:44.966105 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:45.025456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:45.025405 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djhb\" (UniqueName: \"kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb\") pod \"data-science-collector-collector-1\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:45.306169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:45.306069 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:03:45.687291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:45.687260 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:03:45.690146 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:03:45.690111 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1ae603_ed7e_4bb3_a2fa_a56249dfe37b.slice/crio-2ab8b66f88957294552a24ab17b607911487785636fa1bd8757980e8c3c6ac92 WatchSource:0}: Error finding container 2ab8b66f88957294552a24ab17b607911487785636fa1bd8757980e8c3c6ac92: Status 404 returned error can't find the container with id 2ab8b66f88957294552a24ab17b607911487785636fa1bd8757980e8c3c6ac92 Oct 30 21:03:45.692343 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:45.692308 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:03:45.692457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:45.692373 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:03:46.015520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:46.015438 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3b3199-3ad6-444e-9fe4-9be055b5a170" path="/var/lib/kubelet/pods/1f3b3199-3ad6-444e-9fe4-9be055b5a170/volumes" Oct 30 21:03:46.436134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:46.436098 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b","Type":"ContainerStarted","Data":"18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d"} Oct 30 21:03:46.436313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:46.436152 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b","Type":"ContainerStarted","Data":"2ab8b66f88957294552a24ab17b607911487785636fa1bd8757980e8c3c6ac92"} Oct 30 21:03:46.496831 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:46.496793 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:46.561103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:46.561041 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.56102535 podStartE2EDuration="2.56102535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:46.559514509 +0000 UTC m=+1654.667010420" watchObservedRunningTime="2026-10-30 21:03:46.56102535 +0000 UTC m=+1654.668521368" Oct 30 21:03:46.701862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:46.701765 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:54.367038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:54.366961 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:54.589954 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:03:54.589912 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:56.394346 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:03:56.394306 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:04:10.976880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:04:10.976835 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:26.394838 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:04:26.394792 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:04:54.958695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:04:54.958649 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:55.105799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:04:55.105761 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:56.394898 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:04:56.394862 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:05:01.050799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:01.050767 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:01.252656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:01.252623 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:07.274506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:07.274460 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:07.419350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:07.419295 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:09.380888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:09.380849 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:09.527706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:09.527665 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:09.682650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:09.682617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:11.254060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:11.254016 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:26.396025 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:05:26.395946 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:05:31.744446 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:31.744389 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:31.940025 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:31.939984 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:32.082193 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:32.082108 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:32.290326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:32.290278 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:32.433358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:32.433322 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:32.590937 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:32.590904 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:37.453191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:37.453146 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:05:37.453676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:37.453369 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" containerName="otc-container" containerID="cri-o://18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d" gracePeriod=30 Oct 30 21:05:37.790271 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:37.790176 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:37.938429 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:37.938375 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:38.144126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:38.144090 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:39.808812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.808784 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:39.898393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898360 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb\" (UniqueName: \"kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb\") pod \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " Oct 30 21:05:39.898610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898446 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert\") pod \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " Oct 30 21:05:39.898610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898502 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal\") pod \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " Oct 30 21:05:39.898610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898521 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs\") pod \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " Oct 30 21:05:39.898610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898547 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs\") pod \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " Oct 30 21:05:39.898610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898585 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs\") pod \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\" (UID: \"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b\") " Oct 30 21:05:39.898951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.898917 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal" pod "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" (UID: "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:39.900753 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.900714 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert" pod "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" (UID: "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:05:39.900753 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.900733 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs" pod "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" (UID: "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:05:39.900918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.900826 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb" pod "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" (UID: "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b"). InnerVolumeSpecName "kube-api-access-5djhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:39.901112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.901092 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs" pod "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" (UID: "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:05:39.901158 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.901110 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs" pod "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" (UID: "df1ae603-ed7e-4bb3-a2fa-a56249dfe37b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:05:39.967381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.967339 4988 generic.go:356] "Generic (PLEG): container finished" podID="df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" containerID="18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d" exitCode=0 Oct 30 21:05:39.967618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.967399 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:39.967618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.967396 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b","Type":"ContainerDied","Data":"18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d"} Oct 30 21:05:39.967618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.967533 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"df1ae603-ed7e-4bb3-a2fa-a56249dfe37b","Type":"ContainerDied","Data":"2ab8b66f88957294552a24ab17b607911487785636fa1bd8757980e8c3c6ac92"} Oct 30 21:05:39.967618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.967567 4988 scope.go:122] "RemoveContainer" containerID="18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d" Oct 30 21:05:39.979670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.979641 4988 scope.go:122] "RemoveContainer" containerID="18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d" Oct 30 21:05:39.980185 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:05:39.980162 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d\": container with ID starting with 18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d not found: ID does not exist" containerID="18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d" Oct 30 21:05:39.980246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.980200 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d"} err="failed to get container status \"18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d\": rpc error: code = NotFound desc = could not find container \"18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d\": container with ID starting with 18f0f960d38c43cd2fd19a8c4d17a6870584b39e6141f9bbf90dc78b8ad8708d not found: ID does not exist" Oct 30 21:05:39.999695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.999664 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djhb\" (UniqueName: \"kubernetes.io/projected/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-kube-api-access-5djhb\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.999695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.999693 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.999885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.999707 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.999885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.999715 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.999885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.999724 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:39.999885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:39.999733 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:05:40.131809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.131770 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:05:40.138179 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.138145 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:05:40.214730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.214640 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:05:40.215096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.215078 4988 cpu_manager.go:414] "containerMap: removing container" podUID="df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" containerName="otc-container" Oct 30 21:05:40.215179 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.215098 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" containerName="otc-container" Oct 30 21:05:40.215254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.215240 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" containerName="otc-container" Oct 30 21:05:40.219677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.219651 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.272084 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.272049 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:05:40.272084 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.272062 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:05:40.274968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.274936 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:05:40.276028 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.276004 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-db539ca7\"" Oct 30 21:05:40.277266 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.277241 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:05:40.372479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.372437 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:05:40.372664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.372519 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:05:40.402814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.402783 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468n5\" (UniqueName: \"kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.402985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.402818 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.402985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.402846 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.402985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.402909 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.403092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.403028 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.403092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.403073 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.504368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.504277 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468n5\" (UniqueName: \"kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.504368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.504319 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.504368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.504347 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.504651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.504375 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.504651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.504458 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.504651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.504518 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.505321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.505289 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.507166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.507140 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.507267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.507148 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.507364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.507346 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.507487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.507467 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.565109 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.565071 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-468n5\" (UniqueName: \"kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5\") pod \"data-science-collector-collector-1\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:40.833108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:40.833020 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:05:41.209561 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:41.209522 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:05:41.241549 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:05:41.241515 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d54d78_9363_4afc_a5bc_abbd79cff225.slice/crio-c2b83f205a7fc82cc2a08503069996e635b2b03f78ea456b5e7867cea9edbc6e WatchSource:0}: Error finding container c2b83f205a7fc82cc2a08503069996e635b2b03f78ea456b5e7867cea9edbc6e: Status 404 returned error can't find the container with id c2b83f205a7fc82cc2a08503069996e635b2b03f78ea456b5e7867cea9edbc6e Oct 30 21:05:41.244025 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:41.243977 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:05:41.244110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:41.244070 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:05:41.980136 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:41.980101 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"67d54d78-9363-4afc-a5bc-abbd79cff225","Type":"ContainerStarted","Data":"8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15"} Oct 30 21:05:41.980553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:41.980150 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"67d54d78-9363-4afc-a5bc-abbd79cff225","Type":"ContainerStarted","Data":"c2b83f205a7fc82cc2a08503069996e635b2b03f78ea456b5e7867cea9edbc6e"} Oct 30 21:05:42.015695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:42.015665 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1ae603-ed7e-4bb3-a2fa-a56249dfe37b" path="/var/lib/kubelet/pods/df1ae603-ed7e-4bb3-a2fa-a56249dfe37b/volumes" Oct 30 21:05:42.098646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:42.098593 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.09858 podStartE2EDuration="2.09858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:42.097641602 +0000 UTC m=+1770.205137515" watchObservedRunningTime="2026-10-30 21:05:42.09858 +0000 UTC m=+1770.206075912" Oct 30 21:05:44.148200 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:44.148158 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:44.292894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:05:44.292855 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:56.394363 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:05:56.394325 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:06:09.582228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:06:09.582191 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:11.595717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:06:11.595681 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:12.191142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:06:12.191114 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:06:12.198142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:06:12.198112 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:06:26.395495 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:06:26.395456 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:06:56.396127 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:06:56.396043 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:07:12.013215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:12.013179 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:26.395746 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:07:26.395706 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:07:36.979710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:36.979666 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:37.126139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:37.126102 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:37.333092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:37.332993 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:37.543540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:37.543499 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:37.689767 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:37.689729 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:37.836995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:37.836958 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:43.037280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:43.037241 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:07:43.037734 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:43.037490 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="67d54d78-9363-4afc-a5bc-abbd79cff225" containerName="otc-container" containerID="cri-o://8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15" gracePeriod=30 Oct 30 21:07:43.378672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:43.378561 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:43.530159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:43.530108 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:43.747398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:43.747355 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:45.390454 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.390426 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.456782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.456694 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs\") pod \"67d54d78-9363-4afc-a5bc-abbd79cff225\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " Oct 30 21:07:45.456782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.456735 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert\") pod \"67d54d78-9363-4afc-a5bc-abbd79cff225\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " Oct 30 21:07:45.457010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.456802 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs\") pod \"67d54d78-9363-4afc-a5bc-abbd79cff225\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " Oct 30 21:07:45.457010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.456821 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal\" (UniqueName: \"kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal\") pod \"67d54d78-9363-4afc-a5bc-abbd79cff225\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " Oct 30 21:07:45.457010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.456869 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5\" (UniqueName: \"kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5\") pod \"67d54d78-9363-4afc-a5bc-abbd79cff225\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " Oct 30 21:07:45.457010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.456924 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs\") pod \"67d54d78-9363-4afc-a5bc-abbd79cff225\" (UID: \"67d54d78-9363-4afc-a5bc-abbd79cff225\") " Oct 30 21:07:45.457401 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.457370 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal" pod "67d54d78-9363-4afc-a5bc-abbd79cff225" (UID: "67d54d78-9363-4afc-a5bc-abbd79cff225"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:45.459214 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.459183 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5" pod "67d54d78-9363-4afc-a5bc-abbd79cff225" (UID: "67d54d78-9363-4afc-a5bc-abbd79cff225"). InnerVolumeSpecName "kube-api-access-468n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:45.459334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.459215 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs" pod "67d54d78-9363-4afc-a5bc-abbd79cff225" (UID: "67d54d78-9363-4afc-a5bc-abbd79cff225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:45.459334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.459248 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs" pod "67d54d78-9363-4afc-a5bc-abbd79cff225" (UID: "67d54d78-9363-4afc-a5bc-abbd79cff225"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:45.459603 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.459577 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert" pod "67d54d78-9363-4afc-a5bc-abbd79cff225" (UID: "67d54d78-9363-4afc-a5bc-abbd79cff225"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:45.459700 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.459619 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs" pod "67d54d78-9363-4afc-a5bc-abbd79cff225" (UID: "67d54d78-9363-4afc-a5bc-abbd79cff225"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:45.558587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.558547 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.558587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.558584 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.558828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.558597 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.558828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.558611 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67d54d78-9363-4afc-a5bc-abbd79cff225-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.558828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.558626 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-468n5\" (UniqueName: \"kubernetes.io/projected/67d54d78-9363-4afc-a5bc-abbd79cff225-kube-api-access-468n5\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.558828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.558655 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d54d78-9363-4afc-a5bc-abbd79cff225-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.569092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.569055 4988 generic.go:356] "Generic (PLEG): container finished" podID="67d54d78-9363-4afc-a5bc-abbd79cff225" containerID="8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15" exitCode=0 Oct 30 21:07:45.569316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.569109 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.569316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.569153 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"67d54d78-9363-4afc-a5bc-abbd79cff225","Type":"ContainerDied","Data":"8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15"} Oct 30 21:07:45.569316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.569215 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"67d54d78-9363-4afc-a5bc-abbd79cff225","Type":"ContainerDied","Data":"c2b83f205a7fc82cc2a08503069996e635b2b03f78ea456b5e7867cea9edbc6e"} Oct 30 21:07:45.569316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.569246 4988 scope.go:122] "RemoveContainer" containerID="8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15" Oct 30 21:07:45.578509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.578484 4988 scope.go:122] "RemoveContainer" containerID="8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15" Oct 30 21:07:45.578831 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:07:45.578810 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15\": container with ID starting with 8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15 not found: ID does not exist" containerID="8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15" Oct 30 21:07:45.578903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.578843 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15"} err="failed to get container status \"8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15\": rpc error: code = NotFound desc = could not find container \"8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15\": container with ID starting with 8e6e3a4165b6a77cf1a17aee845bcbdc6a995f839257afa63aa4006bdd0c3e15 not found: ID does not exist" Oct 30 21:07:45.735791 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.735697 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:07:45.740585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.740557 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:07:45.824996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.824961 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:07:45.825373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.825359 4988 cpu_manager.go:414] "containerMap: removing container" podUID="67d54d78-9363-4afc-a5bc-abbd79cff225" containerName="otc-container" Oct 30 21:07:45.825442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.825374 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d54d78-9363-4afc-a5bc-abbd79cff225" containerName="otc-container" Oct 30 21:07:45.825482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.825447 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d54d78-9363-4afc-a5bc-abbd79cff225" containerName="otc-container" Oct 30 21:07:45.829624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.829607 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.882776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.882742 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:07:45.883166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.883147 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:07:45.883249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.883231 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:07:45.883304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.883291 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:07:45.883403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.883385 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:07:45.883501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.883482 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:07:45.888846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.888809 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:07:45.963578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.963544 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.963578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.963577 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.963795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.963600 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.963795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.963650 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.963795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.963775 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbgjm\" (UniqueName: \"kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:45.963905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:45.963807 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.015056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.014978 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d54d78-9363-4afc-a5bc-abbd79cff225" path="/var/lib/kubelet/pods/67d54d78-9363-4afc-a5bc-abbd79cff225/volumes" Oct 30 21:07:46.064499 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.064460 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbgjm\" (UniqueName: \"kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.064658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.064503 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.064658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.064561 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.064658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.064575 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.064658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.064596 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.064658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.064610 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.065468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.065399 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.067291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.067261 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.067291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.067290 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.067525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.067327 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.067525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.067511 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.073117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.073088 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:46.125898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.125864 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbgjm\" (UniqueName: \"kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm\") pod \"data-science-collector-collector-1\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.141938 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:07:46.524800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.524765 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:07:46.527978 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:07:46.527949 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd856e27e_506e_445d_8d33_c425e1ed914c.slice/crio-7a7f539f8f9b311ed18f4143e6bfc6251ab1dbd34815b8ec904a04015c3ad654 WatchSource:0}: Error finding container 7a7f539f8f9b311ed18f4143e6bfc6251ab1dbd34815b8ec904a04015c3ad654: Status 404 returned error can't find the container with id 7a7f539f8f9b311ed18f4143e6bfc6251ab1dbd34815b8ec904a04015c3ad654 Oct 30 21:07:46.530208 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.530174 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:07:46.530274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.530234 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:07:46.575979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:46.575943 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d856e27e-506e-445d-8d33-c425e1ed914c","Type":"ContainerStarted","Data":"7a7f539f8f9b311ed18f4143e6bfc6251ab1dbd34815b8ec904a04015c3ad654"} Oct 30 21:07:47.582577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:47.582538 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d856e27e-506e-445d-8d33-c425e1ed914c","Type":"ContainerStarted","Data":"36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8"} Oct 30 21:07:47.702316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:47.702258 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.7022430269999997 podStartE2EDuration="2.702243027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:47.701579896 +0000 UTC m=+1895.809075841" watchObservedRunningTime="2026-10-30 21:07:47.702243027 +0000 UTC m=+1895.809738983" Oct 30 21:07:51.114936 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:51.114897 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:56.163597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:56.163558 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:56.394952 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:07:56.394919 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:07:57.234323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:07:57.234284 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:12.305154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:12.305108 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:23.252808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:23.252711 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:23.399749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:23.399695 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:23.602899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:23.602807 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:23.759361 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:23.759321 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:23.916230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:23.916192 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:24.062222 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:24.062178 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:26.395798 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:08:26.395760 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:08:28.950073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:28.950036 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:08:28.950524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:28.950275 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d856e27e-506e-445d-8d33-c425e1ed914c" containerName="otc-container" containerID="cri-o://36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8" gracePeriod=30 Oct 30 21:08:29.286358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:29.286253 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:29.484605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:29.484564 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:29.647565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:29.647523 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:31.299483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.299453 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:31.341578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341485 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs\") pod \"d856e27e-506e-445d-8d33-c425e1ed914c\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " Oct 30 21:08:31.341578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341530 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm\" (UniqueName: \"kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm\") pod \"d856e27e-506e-445d-8d33-c425e1ed914c\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " Oct 30 21:08:31.341578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341558 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal\") pod \"d856e27e-506e-445d-8d33-c425e1ed914c\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " Oct 30 21:08:31.341872 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341695 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs\") pod \"d856e27e-506e-445d-8d33-c425e1ed914c\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " Oct 30 21:08:31.341872 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341763 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert\") pod \"d856e27e-506e-445d-8d33-c425e1ed914c\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " Oct 30 21:08:31.341872 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341796 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs\") pod \"d856e27e-506e-445d-8d33-c425e1ed914c\" (UID: \"d856e27e-506e-445d-8d33-c425e1ed914c\") " Oct 30 21:08:31.342034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.341997 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal" pod "d856e27e-506e-445d-8d33-c425e1ed914c" (UID: "d856e27e-506e-445d-8d33-c425e1ed914c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:31.342116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.342100 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d856e27e-506e-445d-8d33-c425e1ed914c-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.343886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.343845 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs" pod "d856e27e-506e-445d-8d33-c425e1ed914c" (UID: "d856e27e-506e-445d-8d33-c425e1ed914c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:31.344015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.343903 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm" pod "d856e27e-506e-445d-8d33-c425e1ed914c" (UID: "d856e27e-506e-445d-8d33-c425e1ed914c"). InnerVolumeSpecName "kube-api-access-nbgjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:31.344015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.343939 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs" pod "d856e27e-506e-445d-8d33-c425e1ed914c" (UID: "d856e27e-506e-445d-8d33-c425e1ed914c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:31.344146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.344131 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs" pod "d856e27e-506e-445d-8d33-c425e1ed914c" (UID: "d856e27e-506e-445d-8d33-c425e1ed914c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:31.344186 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.344154 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert" pod "d856e27e-506e-445d-8d33-c425e1ed914c" (UID: "d856e27e-506e-445d-8d33-c425e1ed914c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:31.442761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.442729 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.442761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.442757 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.442761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.442768 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.443005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.442777 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbgjm\" (UniqueName: \"kubernetes.io/projected/d856e27e-506e-445d-8d33-c425e1ed914c-kube-api-access-nbgjm\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.443005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.442786 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d856e27e-506e-445d-8d33-c425e1ed914c-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.790435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.790373 4988 generic.go:356] "Generic (PLEG): container finished" podID="d856e27e-506e-445d-8d33-c425e1ed914c" containerID="36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8" exitCode=0 Oct 30 21:08:31.790646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.790491 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d856e27e-506e-445d-8d33-c425e1ed914c","Type":"ContainerDied","Data":"36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8"} Oct 30 21:08:31.790646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.790544 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d856e27e-506e-445d-8d33-c425e1ed914c","Type":"ContainerDied","Data":"7a7f539f8f9b311ed18f4143e6bfc6251ab1dbd34815b8ec904a04015c3ad654"} Oct 30 21:08:31.790646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.790573 4988 scope.go:122] "RemoveContainer" containerID="36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8" Oct 30 21:08:31.790752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.790739 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:31.800493 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.800466 4988 scope.go:122] "RemoveContainer" containerID="36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8" Oct 30 21:08:31.800803 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:08:31.800786 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8\": container with ID starting with 36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8 not found: ID does not exist" containerID="36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8" Oct 30 21:08:31.800864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.800813 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8"} err="failed to get container status \"36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8\": rpc error: code = NotFound desc = could not find container \"36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8\": container with ID starting with 36bb92f19def0023e02b911f8cde53ac2e938ccf513ef1ec13df851cbdd0dbe8 not found: ID does not exist" Oct 30 21:08:31.958293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:31.958256 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:08:32.006903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.006870 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:08:32.016587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.016552 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d856e27e-506e-445d-8d33-c425e1ed914c" path="/var/lib/kubelet/pods/d856e27e-506e-445d-8d33-c425e1ed914c/volumes" Oct 30 21:08:32.046866 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.046786 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:08:32.047236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.047221 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d856e27e-506e-445d-8d33-c425e1ed914c" containerName="otc-container" Oct 30 21:08:32.047279 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.047238 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d856e27e-506e-445d-8d33-c425e1ed914c" containerName="otc-container" Oct 30 21:08:32.047318 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.047305 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d856e27e-506e-445d-8d33-c425e1ed914c" containerName="otc-container" Oct 30 21:08:32.051554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.051523 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.104474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.104438 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:08:32.104474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.104456 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:08:32.104830 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.104814 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:08:32.108960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.108909 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:08:32.110930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.110905 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:08:32.149425 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.149384 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.149625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.149462 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxg6\" (UniqueName: \"kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.149625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.149563 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.149625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.149611 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.149778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.149698 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.149778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.149732 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.205321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.205264 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:08:32.205321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.205291 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:08:32.251084 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251045 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.251287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251103 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.251287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251134 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.251287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251154 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.251287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251204 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.251287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251226 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxg6\" (UniqueName: \"kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.251986 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.251951 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.253818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.253789 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.253923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.253905 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.253989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.253966 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.254076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.254060 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.312760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.312685 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxg6\" (UniqueName: \"kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6\") pod \"data-science-collector-collector-1\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.364436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.364380 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:08:32.746165 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.746122 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:08:32.750500 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:08:32.750469 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854b3251_f546_4bd5_b0e7_3f91c0d27689.slice/crio-fe82f16b3918073a4c878a4a1e278204159e18e5c1ec1f9243411a3907401a39 WatchSource:0}: Error finding container fe82f16b3918073a4c878a4a1e278204159e18e5c1ec1f9243411a3907401a39: Status 404 returned error can't find the container with id fe82f16b3918073a4c878a4a1e278204159e18e5c1ec1f9243411a3907401a39 Oct 30 21:08:32.752582 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.752546 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:08:32.752678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.752614 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:08:32.797086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:32.797054 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"854b3251-f546-4bd5-b0e7-3f91c0d27689","Type":"ContainerStarted","Data":"fe82f16b3918073a4c878a4a1e278204159e18e5c1ec1f9243411a3907401a39"} Oct 30 21:08:33.378343 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:33.378300 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:33.534481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:33.534443 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:33.697581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:33.697533 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:33.802957 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:33.802921 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"854b3251-f546-4bd5-b0e7-3f91c0d27689","Type":"ContainerStarted","Data":"467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846"} Oct 30 21:08:33.930094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:33.930037 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.930019407 podStartE2EDuration="1.930019407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:33.928551428 +0000 UTC m=+1942.036047365" watchObservedRunningTime="2026-10-30 21:08:33.930019407 +0000 UTC m=+1942.037515319" Oct 30 21:08:33.945593 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:08:33.945551 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:56.394895 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:08:56.394851 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:09:12.648651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:09:12.648618 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.395604 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:09:26.395564 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:09:50.559982 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:09:50.559941 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:56.395759 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:09:56.395722 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:10:12.995081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:10:12.995044 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:26.396014 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:10:26.395969 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:10:56.394867 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:10:56.394827 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:11:09.253013 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:11:09.252970 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:12.221133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:11:12.221100 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:11:12.229578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:11:12.229552 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:11:13.335292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:11:13.335256 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:26.395242 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:11:26.395157 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:11:42.751531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:11:42.751490 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:56.394634 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:11:56.394596 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:12:08.095040 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:08.094993 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:13.734814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:13.734771 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:14.008111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:14.008010 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:14.151896 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:14.151857 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:14.295864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:14.295767 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:20.507498 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:20.507451 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:20.650993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:20.650947 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:22.395935 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:22.395900 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:26.393952 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:12:26.393912 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:12:28.310162 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:28.310110 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:28.461997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:28.461961 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:44.745546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:44.745493 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:44.934814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:44.934775 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:45.076155 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:45.076064 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:45.218469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:45.218432 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:45.362461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:45.362364 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:45.506988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:45.506950 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:45.648787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:45.648748 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:50.519081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:50.519042 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:12:50.519540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:50.519227 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="854b3251-f546-4bd5-b0e7-3f91c0d27689" containerName="otc-container" containerID="cri-o://467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846" gracePeriod=30 Oct 30 21:12:50.850226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:50.850127 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:51.040888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:51.040848 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:51.207745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:51.207708 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:52.867261 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.867227 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:52.897340 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.897308 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs\") pod \"854b3251-f546-4bd5-b0e7-3f91c0d27689\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " Oct 30 21:12:52.897541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.897366 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert\") pod \"854b3251-f546-4bd5-b0e7-3f91c0d27689\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " Oct 30 21:12:52.897541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.897431 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal\" (UniqueName: \"kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal\") pod \"854b3251-f546-4bd5-b0e7-3f91c0d27689\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " Oct 30 21:12:52.897541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.897467 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs\") pod \"854b3251-f546-4bd5-b0e7-3f91c0d27689\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " Oct 30 21:12:52.897541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.897496 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6\" (UniqueName: \"kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6\") pod \"854b3251-f546-4bd5-b0e7-3f91c0d27689\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " Oct 30 21:12:52.897892 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.897866 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal" pod "854b3251-f546-4bd5-b0e7-3f91c0d27689" (UID: "854b3251-f546-4bd5-b0e7-3f91c0d27689"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:52.899634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.899605 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs" pod "854b3251-f546-4bd5-b0e7-3f91c0d27689" (UID: "854b3251-f546-4bd5-b0e7-3f91c0d27689"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:52.899733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.899659 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert" pod "854b3251-f546-4bd5-b0e7-3f91c0d27689" (UID: "854b3251-f546-4bd5-b0e7-3f91c0d27689"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:52.899733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.899700 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs" pod "854b3251-f546-4bd5-b0e7-3f91c0d27689" (UID: "854b3251-f546-4bd5-b0e7-3f91c0d27689"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:52.899828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.899811 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6" pod "854b3251-f546-4bd5-b0e7-3f91c0d27689" (UID: "854b3251-f546-4bd5-b0e7-3f91c0d27689"). InnerVolumeSpecName "kube-api-access-bgxg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:52.998367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.998291 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs\") pod \"854b3251-f546-4bd5-b0e7-3f91c0d27689\" (UID: \"854b3251-f546-4bd5-b0e7-3f91c0d27689\") " Oct 30 21:12:52.998527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.998518 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:52.998570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.998533 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:52.998570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.998544 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/854b3251-f546-4bd5-b0e7-3f91c0d27689-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:52.998570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.998554 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:52.998570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:52.998563 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxg6\" (UniqueName: \"kubernetes.io/projected/854b3251-f546-4bd5-b0e7-3f91c0d27689-kube-api-access-bgxg6\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:53.000375 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.000345 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs" pod "854b3251-f546-4bd5-b0e7-3f91c0d27689" (UID: "854b3251-f546-4bd5-b0e7-3f91c0d27689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:53.039829 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.039787 4988 generic.go:356] "Generic (PLEG): container finished" podID="854b3251-f546-4bd5-b0e7-3f91c0d27689" containerID="467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846" exitCode=0 Oct 30 21:12:53.040015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.039837 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.040015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.039873 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"854b3251-f546-4bd5-b0e7-3f91c0d27689","Type":"ContainerDied","Data":"467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846"} Oct 30 21:12:53.040015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.039934 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"854b3251-f546-4bd5-b0e7-3f91c0d27689","Type":"ContainerDied","Data":"fe82f16b3918073a4c878a4a1e278204159e18e5c1ec1f9243411a3907401a39"} Oct 30 21:12:53.040015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.039956 4988 scope.go:122] "RemoveContainer" containerID="467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846" Oct 30 21:12:53.049101 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.049080 4988 scope.go:122] "RemoveContainer" containerID="467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846" Oct 30 21:12:53.049458 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:12:53.049436 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846\": container with ID starting with 467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846 not found: ID does not exist" containerID="467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846" Oct 30 21:12:53.049525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.049467 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846"} err="failed to get container status \"467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846\": rpc error: code = NotFound desc = could not find container \"467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846\": container with ID starting with 467a1883f1ee8ed7c950da539567d9b069b900e7b5372ed115ee1dcea51fe846 not found: ID does not exist" Oct 30 21:12:53.099396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.099362 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854b3251-f546-4bd5-b0e7-3f91c0d27689-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:12:53.203462 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.203403 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:12:53.208697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.208664 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:12:53.293312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.293236 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:12:53.293652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.293638 4988 cpu_manager.go:414] "containerMap: removing container" podUID="854b3251-f546-4bd5-b0e7-3f91c0d27689" containerName="otc-container" Oct 30 21:12:53.293699 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.293654 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854b3251-f546-4bd5-b0e7-3f91c0d27689" containerName="otc-container" Oct 30 21:12:53.293746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.293737 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="854b3251-f546-4bd5-b0e7-3f91c0d27689" containerName="otc-container" Oct 30 21:12:53.297998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.297979 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.300616 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.300591 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.300747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.300619 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.300747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.300737 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.300840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.300786 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pb2s\" (UniqueName: \"kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.300840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.300812 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.300907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.300838 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.350468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.350366 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:12:53.350670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.350494 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:12:53.350670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.350494 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:12:53.350670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.350495 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-jqj7h\"" Oct 30 21:12:53.350670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.350534 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:12:53.350670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.350586 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:12:53.362466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.362428 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:12:53.401887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.401848 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.401887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.401889 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2s\" (UniqueName: \"kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.402152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.401915 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.402152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.401932 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.402152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.401957 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.402152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.401987 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.402756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.402735 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.404549 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.404525 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.404632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.404554 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.404804 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.404786 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.404842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.404801 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.463030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.462956 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pb2s\" (UniqueName: \"kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s\") pod \"data-science-collector-collector-1\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.610502 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.610404 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:12:53.987065 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.987032 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:12:53.990529 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:12:53.990491 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a66ef0_303f_452f_ade2_30c248452416.slice/crio-ed6b1ba89311b95e8540228bb91cb9bc4f39e58f8c1afbeb8246257869fbec86 WatchSource:0}: Error finding container ed6b1ba89311b95e8540228bb91cb9bc4f39e58f8c1afbeb8246257869fbec86: Status 404 returned error can't find the container with id ed6b1ba89311b95e8540228bb91cb9bc4f39e58f8c1afbeb8246257869fbec86 Oct 30 21:12:53.992970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.992938 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:12:53.993048 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:53.993003 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:12:54.016282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:54.016248 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854b3251-f546-4bd5-b0e7-3f91c0d27689" path="/var/lib/kubelet/pods/854b3251-f546-4bd5-b0e7-3f91c0d27689/volumes" Oct 30 21:12:54.046314 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:54.046266 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02a66ef0-303f-452f-ade2-30c248452416","Type":"ContainerStarted","Data":"ed6b1ba89311b95e8540228bb91cb9bc4f39e58f8c1afbeb8246257869fbec86"} Oct 30 21:12:55.057430 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:55.057368 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02a66ef0-303f-452f-ade2-30c248452416","Type":"ContainerStarted","Data":"29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7"} Oct 30 21:12:55.209512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:12:55.209454 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.209431638 podStartE2EDuration="2.209431638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:55.206489903 +0000 UTC m=+2203.313985815" watchObservedRunningTime="2026-10-30 21:12:55.209431638 +0000 UTC m=+2203.316927557" Oct 30 21:12:56.396494 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:12:56.396456 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:13:14.031617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:13:14.031573 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:26.396031 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:13:26.395979 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:13:46.525314 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:13:46.525275 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:56.396743 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:13:56.396699 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:14:00.494262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:14:00.494218 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.638824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:14:00.638789 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.373724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:14:14.373682 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:26.396094 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:14:26.395991 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:14:56.395527 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:14:56.395487 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:15:14.723055 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:15:14.723019 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:26.396106 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:15:26.396069 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:15:53.805172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:15:53.805070 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:53.999495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:15:53.999453 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:56.395298 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:15:56.395254 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:16:12.250366 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:16:12.250333 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:16:12.260331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:16:12.260307 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:16:15.066016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:16:15.065967 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:21.507646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:16:21.507572 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:26.394931 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:16:26.394896 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:16:56.394560 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:16:56.394509 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:17:15.466293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:15.466246 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:26.395747 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:17:26.395656 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:17:45.424820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:45.424779 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:45.623150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:45.623116 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:52.567748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:52.567715 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:52.786474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:52.786395 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:56.394888 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:17:56.394845 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:17:56.801001 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:56.800896 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:57.001534 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:17:57.001489 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:01.326964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:01.326916 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:11.677835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:11.677787 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:14.425709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.425671 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5"] Oct 30 21:18:14.426225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.425996 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="kube-rbac-proxy" containerID="cri-o://fc7e7f15d66a013016a7e7eb64b108c28cb8d86d9bcf206359a9e4a6f02b30e9" gracePeriod=30 Oct 30 21:18:14.426225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.426046 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="prom-label-proxy" containerID="cri-o://7c528163431571df9a186d44b41e3c57d010105581dc8595069654b9d6b7ca67" gracePeriod=30 Oct 30 21:18:14.426225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.426178 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="method-gate" containerID="cri-o://739875c010f91b5f7d557c0bae7a5559893316a727be60fb0889c88c275a7dea" gracePeriod=30 Oct 30 21:18:14.480281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.480242 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh"] Oct 30 21:18:14.480540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.480513 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="kube-rbac-proxy" containerID="cri-o://6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" gracePeriod=30 Oct 30 21:18:14.480600 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.480550 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="method-gate" containerID="cri-o://01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" gracePeriod=30 Oct 30 21:18:14.480640 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.480520 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="prom-label-proxy" containerID="cri-o://4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" gracePeriod=30 Oct 30 21:18:14.554693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.554662 4988 generic.go:356] "Generic (PLEG): container finished" podID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerID="7c528163431571df9a186d44b41e3c57d010105581dc8595069654b9d6b7ca67" exitCode=0 Oct 30 21:18:14.554693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.554687 4988 generic.go:356] "Generic (PLEG): container finished" podID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerID="739875c010f91b5f7d557c0bae7a5559893316a727be60fb0889c88c275a7dea" exitCode=0 Oct 30 21:18:14.554693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.554697 4988 generic.go:356] "Generic (PLEG): container finished" podID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerID="fc7e7f15d66a013016a7e7eb64b108c28cb8d86d9bcf206359a9e4a6f02b30e9" exitCode=0 Oct 30 21:18:14.554960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.554733 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerDied","Data":"7c528163431571df9a186d44b41e3c57d010105581dc8595069654b9d6b7ca67"} Oct 30 21:18:14.554960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.554807 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerDied","Data":"739875c010f91b5f7d557c0bae7a5559893316a727be60fb0889c88c275a7dea"} Oct 30 21:18:14.554960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.554844 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerDied","Data":"fc7e7f15d66a013016a7e7eb64b108c28cb8d86d9bcf206359a9e4a6f02b30e9"} Oct 30 21:18:14.731801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.731770 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 21:18:14.756715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.756687 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 21:18:14.835455 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835395 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6\" (UniqueName: \"kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6\") pod \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " Oct 30 21:18:14.835665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835474 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca\") pod \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " Oct 30 21:18:14.835665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835509 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config\" (UniqueName: \"kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config\") pod \"2bd1d521-f511-4f19-993d-38c4e069d692\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " Oct 30 21:18:14.835665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835533 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls\" (UniqueName: \"kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls\") pod \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " Oct 30 21:18:14.835665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835580 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca\") pod \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " Oct 30 21:18:14.835665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835623 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config\") pod \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\" (UID: \"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d\") " Oct 30 21:18:14.835665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835664 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn\" (UniqueName: \"kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn\") pod \"2bd1d521-f511-4f19-993d-38c4e069d692\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " Oct 30 21:18:14.835998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835722 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls\" (UniqueName: \"kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls\") pod \"2bd1d521-f511-4f19-993d-38c4e069d692\" (UID: \"2bd1d521-f511-4f19-993d-38c4e069d692\") " Oct 30 21:18:14.835998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835906 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca" pod "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" (UID: "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:14.835998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.835935 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config" pod "2bd1d521-f511-4f19-993d-38c4e069d692" (UID: "2bd1d521-f511-4f19-993d-38c4e069d692"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:14.836168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.836127 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.836168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.836144 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config" pod "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" (UID: "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:14.836398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.836358 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca" pod "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" (UID: "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:14.837897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.837870 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls" pod "2bd1d521-f511-4f19-993d-38c4e069d692" (UID: "2bd1d521-f511-4f19-993d-38c4e069d692"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:14.838111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.838091 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls" pod "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" (UID: "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:14.838153 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.838093 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6" pod "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" (UID: "62d80f20-1d3c-44bd-99c9-3d2ed1949f8d"). InnerVolumeSpecName "kube-api-access-vpxf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:14.838201 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.838185 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn" pod "2bd1d521-f511-4f19-993d-38c4e069d692" (UID: "2bd1d521-f511-4f19-993d-38c4e069d692"). InnerVolumeSpecName "kube-api-access-vjhfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:14.936985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.936950 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjhfn\" (UniqueName: \"kubernetes.io/projected/2bd1d521-f511-4f19-993d-38c4e069d692-kube-api-access-vjhfn\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.936985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.936983 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bd1d521-f511-4f19-993d-38c4e069d692-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.936985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.936993 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxf6\" (UniqueName: \"kubernetes.io/projected/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-kube-api-access-vpxf6\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.937221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.937001 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bd1d521-f511-4f19-993d-38c4e069d692-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.937221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.937010 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.937221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.937018 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:14.937221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:14.937026 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:15.460383 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.460340 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.563483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.563451 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" Oct 30 21:18:15.563483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.563461 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5" event={"ID":"62d80f20-1d3c-44bd-99c9-3d2ed1949f8d","Type":"ContainerDied","Data":"d94abf5ead9115bf2468b112a8a422ebe60df8ef95aa14453ba02c4b79e996f7"} Oct 30 21:18:15.563746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.563539 4988 scope.go:122] "RemoveContainer" containerID="7c528163431571df9a186d44b41e3c57d010105581dc8595069654b9d6b7ca67" Oct 30 21:18:15.566068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566037 4988 generic.go:356] "Generic (PLEG): container finished" podID="2bd1d521-f511-4f19-993d-38c4e069d692" containerID="4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" exitCode=0 Oct 30 21:18:15.566068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566068 4988 generic.go:356] "Generic (PLEG): container finished" podID="2bd1d521-f511-4f19-993d-38c4e069d692" containerID="01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" exitCode=0 Oct 30 21:18:15.566286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566080 4988 generic.go:356] "Generic (PLEG): container finished" podID="2bd1d521-f511-4f19-993d-38c4e069d692" containerID="6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" exitCode=0 Oct 30 21:18:15.566286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566106 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" Oct 30 21:18:15.566286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566125 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerDied","Data":"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661"} Oct 30 21:18:15.566286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566211 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerDied","Data":"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c"} Oct 30 21:18:15.566286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566253 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerDied","Data":"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0"} Oct 30 21:18:15.566551 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.566289 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh" event={"ID":"2bd1d521-f511-4f19-993d-38c4e069d692","Type":"ContainerDied","Data":"2fa3de3c1fb0307fe5c96419d17416ebef6911c18390476f89ca2462f30d51d3"} Oct 30 21:18:15.574464 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.574440 4988 scope.go:122] "RemoveContainer" containerID="739875c010f91b5f7d557c0bae7a5559893316a727be60fb0889c88c275a7dea" Oct 30 21:18:15.583210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.583148 4988 scope.go:122] "RemoveContainer" containerID="fc7e7f15d66a013016a7e7eb64b108c28cb8d86d9bcf206359a9e4a6f02b30e9" Oct 30 21:18:15.592103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.592078 4988 scope.go:122] "RemoveContainer" containerID="4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" Oct 30 21:18:15.600678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.600658 4988 scope.go:122] "RemoveContainer" containerID="01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" Oct 30 21:18:15.609565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.609549 4988 scope.go:122] "RemoveContainer" containerID="6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" Oct 30 21:18:15.618306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.618282 4988 scope.go:122] "RemoveContainer" containerID="4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" Oct 30 21:18:15.618668 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:15.618647 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": container with ID starting with 4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661 not found: ID does not exist" containerID="4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" Oct 30 21:18:15.618736 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.618680 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661"} err="failed to get container status \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": rpc error: code = NotFound desc = could not find container \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": container with ID starting with 4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661 not found: ID does not exist" Oct 30 21:18:15.618736 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.618698 4988 scope.go:122] "RemoveContainer" containerID="01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" Oct 30 21:18:15.618986 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:15.618972 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": container with ID starting with 01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c not found: ID does not exist" containerID="01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" Oct 30 21:18:15.619026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.618991 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c"} err="failed to get container status \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": rpc error: code = NotFound desc = could not find container \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": container with ID starting with 01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c not found: ID does not exist" Oct 30 21:18:15.619026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619004 4988 scope.go:122] "RemoveContainer" containerID="6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" Oct 30 21:18:15.619221 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:15.619207 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": container with ID starting with 6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0 not found: ID does not exist" containerID="6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" Oct 30 21:18:15.619263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619226 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0"} err="failed to get container status \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": rpc error: code = NotFound desc = could not find container \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": container with ID starting with 6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0 not found: ID does not exist" Oct 30 21:18:15.619263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619239 4988 scope.go:122] "RemoveContainer" containerID="4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" Oct 30 21:18:15.619531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619515 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661"} err="failed to get container status \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": rpc error: code = NotFound desc = could not find container \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": container with ID starting with 4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661 not found: ID does not exist" Oct 30 21:18:15.619580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619532 4988 scope.go:122] "RemoveContainer" containerID="01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" Oct 30 21:18:15.619746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619731 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c"} err="failed to get container status \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": rpc error: code = NotFound desc = could not find container \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": container with ID starting with 01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c not found: ID does not exist" Oct 30 21:18:15.619789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619746 4988 scope.go:122] "RemoveContainer" containerID="6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" Oct 30 21:18:15.619989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619974 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0"} err="failed to get container status \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": rpc error: code = NotFound desc = could not find container \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": container with ID starting with 6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0 not found: ID does not exist" Oct 30 21:18:15.620035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.619989 4988 scope.go:122] "RemoveContainer" containerID="4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661" Oct 30 21:18:15.620174 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.620159 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661"} err="failed to get container status \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": rpc error: code = NotFound desc = could not find container \"4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661\": container with ID starting with 4db5a157fe8b6719ff6216c342d7cade83bc26562bb32526d112414b176e0661 not found: ID does not exist" Oct 30 21:18:15.620222 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.620173 4988 scope.go:122] "RemoveContainer" containerID="01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c" Oct 30 21:18:15.620382 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.620360 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c"} err="failed to get container status \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": rpc error: code = NotFound desc = could not find container \"01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c\": container with ID starting with 01362ea24c7f750378cba183e9227c66c85238160ddd6fe6d91646b720612d0c not found: ID does not exist" Oct 30 21:18:15.620442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.620386 4988 scope.go:122] "RemoveContainer" containerID="6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0" Oct 30 21:18:15.620628 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.620612 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0"} err="failed to get container status \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": rpc error: code = NotFound desc = could not find container \"6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0\": container with ID starting with 6406b696a9fb9a7f04e56154d2cc22da5a2d1aef782870a63d41f86877295de0 not found: ID does not exist" Oct 30 21:18:15.653905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.653860 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.731372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.731271 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5"] Oct 30 21:18:15.748263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.748228 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-csqs5"] Oct 30 21:18:15.754617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.754590 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:18:15.754878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.754856 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="prometheus" containerID="cri-o://917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" gracePeriod=600 Oct 30 21:18:15.754937 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.754871 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="thanos-sidecar" containerID="cri-o://967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" gracePeriod=600 Oct 30 21:18:15.754983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.754871 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="config-reloader" containerID="cri-o://8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" gracePeriod=600 Oct 30 21:18:15.755983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.755958 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:18:15.756193 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.756170 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="alertmanager" containerID="cri-o://28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64" gracePeriod=120 Oct 30 21:18:15.756278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.756245 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="config-reloader" containerID="cri-o://047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293" gracePeriod=120 Oct 30 21:18:15.762981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.762944 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:18:15.801782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.801746 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.842729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.842704 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh"] Oct 30 21:18:15.850504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.850455 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-xfhhh"] Oct 30 21:18:15.953911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:15.953868 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:16.008230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.008202 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:18:16.018745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.018714 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" path="/var/lib/kubelet/pods/2bd1d521-f511-4f19-993d-38c4e069d692/volumes" Oct 30 21:18:16.019195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.019182 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" path="/var/lib/kubelet/pods/62d80f20-1d3c-44bd-99c9-3d2ed1949f8d/volumes" Oct 30 21:18:16.103626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.103583 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:16.148092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148052 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.148273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148120 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.148273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148181 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.148273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148229 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.148273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148263 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.148510 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148466 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.148573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148503 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:16.148731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148703 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:16.148856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148718 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.149113 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.148964 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.149113 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.149001 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.149113 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.149027 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.149113 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.149077 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.149353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.149149 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:16.149791 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.149768 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.149901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.149809 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file\") pod \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\" (UID: \"deeeab09-eb4d-486c-8d3e-b9beca0b768b\") " Oct 30 21:18:16.150301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.150127 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.150301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.150153 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.150301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.150171 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/deeeab09-eb4d-486c-8d3e-b9beca0b768b-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.151106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.151077 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:16.151220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.151102 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:16.151220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.151162 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:16.152692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.152668 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:16.152795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.152756 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:16.153560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.153517 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:16.153670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.153572 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "kube-api-access-4mw92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:16.153670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.153635 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:16.155557 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.155538 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "pvc-e73afc79-aa37-49a0-8744-633c03cad1f4". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:18:16.164775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.164730 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config" pod "deeeab09-eb4d-486c-8d3e-b9beca0b768b" (UID: "deeeab09-eb4d-486c-8d3e-b9beca0b768b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:16.251764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251672 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.251764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251715 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.251764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251733 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.251764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251751 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.252021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251768 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.252021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251783 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/deeeab09-eb4d-486c-8d3e-b9beca0b768b-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.252021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251825 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:18:16.252021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251841 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mw92\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-kube-api-access-4mw92\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.252021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251856 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/deeeab09-eb4d-486c-8d3e-b9beca0b768b-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.252021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.251869 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/deeeab09-eb4d-486c-8d3e-b9beca0b768b-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:16.267866 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.267840 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e73afc79-aa37-49a0-8744-633c03cad1f4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:18:16.352706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.352663 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-468025a65b88095496ffee260c47731d5a0a7d1245692f39a473b974fd0f0891\"" Oct 30 21:18:16.573993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.573912 4988 generic.go:356] "Generic (PLEG): container finished" podID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerID="047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293" exitCode=0 Oct 30 21:18:16.574390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.573986 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerDied","Data":"047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293"} Oct 30 21:18:16.576197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576170 4988 generic.go:356] "Generic (PLEG): container finished" podID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerID="967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" exitCode=0 Oct 30 21:18:16.576197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576194 4988 generic.go:356] "Generic (PLEG): container finished" podID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerID="8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" exitCode=0 Oct 30 21:18:16.576197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576201 4988 generic.go:356] "Generic (PLEG): container finished" podID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerID="917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" exitCode=0 Oct 30 21:18:16.576460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576247 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:18:16.576460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576251 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerDied","Data":"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74"} Oct 30 21:18:16.576460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576327 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerDied","Data":"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1"} Oct 30 21:18:16.576460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576364 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerDied","Data":"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250"} Oct 30 21:18:16.576460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576398 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"deeeab09-eb4d-486c-8d3e-b9beca0b768b","Type":"ContainerDied","Data":"7b3639a7329ec048e167eda4d50095b65235cb0d852c7e3033acd67510466643"} Oct 30 21:18:16.576460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.576399 4988 scope.go:122] "RemoveContainer" containerID="967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" Oct 30 21:18:16.585994 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.585969 4988 scope.go:122] "RemoveContainer" containerID="8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" Oct 30 21:18:16.594738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.594710 4988 scope.go:122] "RemoveContainer" containerID="917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" Oct 30 21:18:16.603894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.603871 4988 scope.go:122] "RemoveContainer" containerID="78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021" Oct 30 21:18:16.613161 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.613137 4988 scope.go:122] "RemoveContainer" containerID="967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" Oct 30 21:18:16.613598 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:16.613578 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": container with ID starting with 967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74 not found: ID does not exist" containerID="967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" Oct 30 21:18:16.613665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.613608 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74"} err="failed to get container status \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": rpc error: code = NotFound desc = could not find container \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": container with ID starting with 967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74 not found: ID does not exist" Oct 30 21:18:16.613665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.613626 4988 scope.go:122] "RemoveContainer" containerID="8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" Oct 30 21:18:16.613938 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:16.613919 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": container with ID starting with 8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1 not found: ID does not exist" containerID="8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" Oct 30 21:18:16.613989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.613945 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1"} err="failed to get container status \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": rpc error: code = NotFound desc = could not find container \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": container with ID starting with 8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1 not found: ID does not exist" Oct 30 21:18:16.613989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.613961 4988 scope.go:122] "RemoveContainer" containerID="917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" Oct 30 21:18:16.614183 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:16.614165 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": container with ID starting with 917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250 not found: ID does not exist" containerID="917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" Oct 30 21:18:16.614230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.614186 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250"} err="failed to get container status \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": rpc error: code = NotFound desc = could not find container \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": container with ID starting with 917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250 not found: ID does not exist" Oct 30 21:18:16.614230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.614201 4988 scope.go:122] "RemoveContainer" containerID="78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021" Oct 30 21:18:16.614521 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:16.614488 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": container with ID starting with 78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021 not found: ID does not exist" containerID="78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021" Oct 30 21:18:16.614595 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.614531 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021"} err="failed to get container status \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": rpc error: code = NotFound desc = could not find container \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": container with ID starting with 78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021 not found: ID does not exist" Oct 30 21:18:16.614595 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.614552 4988 scope.go:122] "RemoveContainer" containerID="967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" Oct 30 21:18:16.614812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.614791 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74"} err="failed to get container status \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": rpc error: code = NotFound desc = could not find container \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": container with ID starting with 967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74 not found: ID does not exist" Oct 30 21:18:16.614812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.614812 4988 scope.go:122] "RemoveContainer" containerID="8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" Oct 30 21:18:16.615073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615051 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1"} err="failed to get container status \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": rpc error: code = NotFound desc = could not find container \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": container with ID starting with 8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1 not found: ID does not exist" Oct 30 21:18:16.615149 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615075 4988 scope.go:122] "RemoveContainer" containerID="917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" Oct 30 21:18:16.615378 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615351 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250"} err="failed to get container status \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": rpc error: code = NotFound desc = could not find container \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": container with ID starting with 917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250 not found: ID does not exist" Oct 30 21:18:16.615378 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615376 4988 scope.go:122] "RemoveContainer" containerID="78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021" Oct 30 21:18:16.615686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615666 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021"} err="failed to get container status \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": rpc error: code = NotFound desc = could not find container \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": container with ID starting with 78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021 not found: ID does not exist" Oct 30 21:18:16.615686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615685 4988 scope.go:122] "RemoveContainer" containerID="967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74" Oct 30 21:18:16.615936 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615921 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74"} err="failed to get container status \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": rpc error: code = NotFound desc = could not find container \"967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74\": container with ID starting with 967f7e8a0bb1b11edf6ce9897779ad412f5fa71a77a18edb0dd51a21027f3f74 not found: ID does not exist" Oct 30 21:18:16.615979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.615937 4988 scope.go:122] "RemoveContainer" containerID="8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1" Oct 30 21:18:16.616194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.616172 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1"} err="failed to get container status \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": rpc error: code = NotFound desc = could not find container \"8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1\": container with ID starting with 8146935a2cc0d8090859cd6c8a3e55a9b8e29b6d1f013f54244029fe36437ee1 not found: ID does not exist" Oct 30 21:18:16.616194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.616194 4988 scope.go:122] "RemoveContainer" containerID="917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250" Oct 30 21:18:16.616480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.616462 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250"} err="failed to get container status \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": rpc error: code = NotFound desc = could not find container \"917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250\": container with ID starting with 917fcd79c7b1e8ebc4937e41f85627d70f41bd300203f673da322c5e01f6a250 not found: ID does not exist" Oct 30 21:18:16.616537 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.616481 4988 scope.go:122] "RemoveContainer" containerID="78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021" Oct 30 21:18:16.616732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.616711 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021"} err="failed to get container status \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": rpc error: code = NotFound desc = could not find container \"78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021\": container with ID starting with 78d7892d066f457f4239b6fbe225f2297fd9e8bdd38d9bea690a6d3d0f792021 not found: ID does not exist" Oct 30 21:18:16.741926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.741892 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:18:16.746198 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:16.746160 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:18:17.627262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:17.627223 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:17.858522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:17.858433 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:18.015889 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.015852 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" path="/var/lib/kubelet/pods/deeeab09-eb4d-486c-8d3e-b9beca0b768b/volumes" Oct 30 21:18:18.104884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.104850 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:18:18.105095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.105063 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="02a66ef0-303f-452f-ade2-30c248452416" containerName="otc-container" containerID="cri-o://29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7" gracePeriod=30 Oct 30 21:18:18.144335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.144291 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:18.319404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.319372 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:18.363152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.363121 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:18.470321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470284 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal\" (UniqueName: \"kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal\") pod \"02a66ef0-303f-452f-ade2-30c248452416\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " Oct 30 21:18:18.470566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470352 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s\" (UniqueName: \"kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s\") pod \"02a66ef0-303f-452f-ade2-30c248452416\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " Oct 30 21:18:18.470566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470456 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs\") pod \"02a66ef0-303f-452f-ade2-30c248452416\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " Oct 30 21:18:18.470566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470496 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert\") pod \"02a66ef0-303f-452f-ade2-30c248452416\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " Oct 30 21:18:18.470566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470532 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs\") pod \"02a66ef0-303f-452f-ade2-30c248452416\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " Oct 30 21:18:18.470796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470567 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs\") pod \"02a66ef0-303f-452f-ade2-30c248452416\" (UID: \"02a66ef0-303f-452f-ade2-30c248452416\") " Oct 30 21:18:18.470846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.470817 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal" pod "02a66ef0-303f-452f-ade2-30c248452416" (UID: "02a66ef0-303f-452f-ade2-30c248452416"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:18.472757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.472727 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs" pod "02a66ef0-303f-452f-ade2-30c248452416" (UID: "02a66ef0-303f-452f-ade2-30c248452416"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:18.472899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.472774 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs" pod "02a66ef0-303f-452f-ade2-30c248452416" (UID: "02a66ef0-303f-452f-ade2-30c248452416"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:18.472899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.472796 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert" pod "02a66ef0-303f-452f-ade2-30c248452416" (UID: "02a66ef0-303f-452f-ade2-30c248452416"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:18.473014 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.472998 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s" pod "02a66ef0-303f-452f-ade2-30c248452416" (UID: "02a66ef0-303f-452f-ade2-30c248452416"). InnerVolumeSpecName "kube-api-access-7pb2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:18.473076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.473063 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs" pod "02a66ef0-303f-452f-ade2-30c248452416" (UID: "02a66ef0-303f-452f-ade2-30c248452416"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:18.566630 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.566588 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:18.571544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.571499 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:18.571544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.571542 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/02a66ef0-303f-452f-ade2-30c248452416-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:18.571544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.571551 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pb2s\" (UniqueName: \"kubernetes.io/projected/02a66ef0-303f-452f-ade2-30c248452416-kube-api-access-7pb2s\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:18.571771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.571562 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:18.571771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.571572 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:18.571771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.571581 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a66ef0-303f-452f-ade2-30c248452416-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:18.590745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.590702 4988 generic.go:356] "Generic (PLEG): container finished" podID="02a66ef0-303f-452f-ade2-30c248452416" containerID="29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7" exitCode=0 Oct 30 21:18:18.590903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.590756 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:18.590903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.590785 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02a66ef0-303f-452f-ade2-30c248452416","Type":"ContainerDied","Data":"29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7"} Oct 30 21:18:18.590903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.590848 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"02a66ef0-303f-452f-ade2-30c248452416","Type":"ContainerDied","Data":"ed6b1ba89311b95e8540228bb91cb9bc4f39e58f8c1afbeb8246257869fbec86"} Oct 30 21:18:18.590903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.590872 4988 scope.go:122] "RemoveContainer" containerID="29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7" Oct 30 21:18:18.600119 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.600097 4988 scope.go:122] "RemoveContainer" containerID="29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7" Oct 30 21:18:18.600474 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:18.600451 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7\": container with ID starting with 29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7 not found: ID does not exist" containerID="29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7" Oct 30 21:18:18.600533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.600483 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7"} err="failed to get container status \"29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7\": rpc error: code = NotFound desc = could not find container \"29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7\": container with ID starting with 29e518db50ae50d78f5e01afc01271f79e02227079c92b515a5605a5c7d198d7 not found: ID does not exist" Oct 30 21:18:18.755118 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.755024 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:18:18.758822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:18.758793 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:18:19.724906 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:19.724871 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:19.868210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:19.868170 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:20.018106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:20.018005 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a66ef0-303f-452f-ade2-30c248452416" path="/var/lib/kubelet/pods/02a66ef0-303f-452f-ade2-30c248452416/volumes" Oct 30 21:18:20.023110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:20.023075 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:20.204810 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:20.204769 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:20.674220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:20.674177 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.55:9093/-/ready\": dial tcp 10.132.0.55:9093: connect: connection refused" Oct 30 21:18:25.674237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:25.674186 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.55:9093/-/ready\": dial tcp 10.132.0.55:9093: connect: connection refused" Oct 30 21:18:25.998393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:25.998356 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:18:26.136744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136703 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.136744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136740 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.136962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136801 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.136962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136827 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.136962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136883 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.136962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136934 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.137180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.136977 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume\") pod \"d96216c3-f3c3-4293-a4db-f793381e1a51\" (UID: \"d96216c3-f3c3-4293-a4db-f793381e1a51\") " Oct 30 21:18:26.138454 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.137494 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:26.139865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.139395 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:26.139984 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.139862 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:26.139984 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.139940 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:26.140350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.140314 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:26.141176 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.141152 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "kube-api-access-nlj89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:26.146644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.146508 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:26.155329 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.155118 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config" pod "d96216c3-f3c3-4293-a4db-f793381e1a51" (UID: "d96216c3-f3c3-4293-a4db-f793381e1a51"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:26.238575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238483 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.238575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238513 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.238575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238548 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.238575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238560 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/d96216c3-f3c3-4293-a4db-f793381e1a51-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.238575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238571 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d96216c3-f3c3-4293-a4db-f793381e1a51-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.238575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238580 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlj89\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-kube-api-access-nlj89\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.238942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.238590 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d96216c3-f3c3-4293-a4db-f793381e1a51-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.340177 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.340137 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:26.396238 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:26.396199 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:18:26.519567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.519467 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:26.631157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.631122 4988 generic.go:356] "Generic (PLEG): container finished" podID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerID="28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64" exitCode=0 Oct 30 21:18:26.631316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.631184 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:18:26.631316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.631202 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerDied","Data":"28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64"} Oct 30 21:18:26.631316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.631262 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"d96216c3-f3c3-4293-a4db-f793381e1a51","Type":"ContainerDied","Data":"dcb74adf334962fda088c1f1bd4cce5263a879d3f85cd9d8cb1735786f076261"} Oct 30 21:18:26.631316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.631291 4988 scope.go:122] "RemoveContainer" containerID="047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293" Oct 30 21:18:26.639921 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.639900 4988 scope.go:122] "RemoveContainer" containerID="28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64" Oct 30 21:18:26.648907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.648881 4988 scope.go:122] "RemoveContainer" containerID="6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32" Oct 30 21:18:26.657030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.657008 4988 scope.go:122] "RemoveContainer" containerID="047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293" Oct 30 21:18:26.657362 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:26.657338 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293\": container with ID starting with 047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293 not found: ID does not exist" containerID="047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293" Oct 30 21:18:26.657405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.657374 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293"} err="failed to get container status \"047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293\": rpc error: code = NotFound desc = could not find container \"047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293\": container with ID starting with 047eac09d8bcd84895dae8522178c3487bd722dc6c69f23c7a5e4ea37a267293 not found: ID does not exist" Oct 30 21:18:26.657405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.657393 4988 scope.go:122] "RemoveContainer" containerID="28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64" Oct 30 21:18:26.657669 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:26.657651 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64\": container with ID starting with 28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64 not found: ID does not exist" containerID="28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64" Oct 30 21:18:26.657708 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.657673 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64"} err="failed to get container status \"28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64\": rpc error: code = NotFound desc = could not find container \"28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64\": container with ID starting with 28b45064242845757442566bfdf750b260358c636a914f63cefa8b7aaf389c64 not found: ID does not exist" Oct 30 21:18:26.657708 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.657685 4988 scope.go:122] "RemoveContainer" containerID="6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32" Oct 30 21:18:26.657893 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:26.657876 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32\": container with ID starting with 6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32 not found: ID does not exist" containerID="6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32" Oct 30 21:18:26.657936 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.657898 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32"} err="failed to get container status \"6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32\": rpc error: code = NotFound desc = could not find container \"6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32\": container with ID starting with 6e9ab9a27a0b7af012475245b331747e333c11f0a12f502cacfc72d4f8d0fd32 not found: ID does not exist" Oct 30 21:18:26.672652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.672617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:26.797032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.796997 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:18:26.802017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:26.801986 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:18:28.015085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.015052 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" path="/var/lib/kubelet/pods/d96216c3-f3c3-4293-a4db-f793381e1a51/volumes" Oct 30 21:18:28.542405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.542361 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.839724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.839640 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:18:28.839978 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.839965 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="kube-rbac-proxy" Oct 30 21:18:28.840019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.839978 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="kube-rbac-proxy" Oct 30 21:18:28.840019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.839990 4988 cpu_manager.go:414] "containerMap: removing container" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="prometheus" Oct 30 21:18:28.840019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.839995 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="prometheus" Oct 30 21:18:28.840019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840004 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="config-reloader" Oct 30 21:18:28.840019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840010 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="config-reloader" Oct 30 21:18:28.840019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840018 4988 cpu_manager.go:414] "containerMap: removing container" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="config-reloader" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840023 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="config-reloader" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840029 4988 cpu_manager.go:414] "containerMap: removing container" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="thanos-sidecar" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840033 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="thanos-sidecar" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840041 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="method-gate" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840046 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="method-gate" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840053 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="init-config-reloader" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840057 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="init-config-reloader" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840065 4988 cpu_manager.go:414] "containerMap: removing container" podUID="02a66ef0-303f-452f-ade2-30c248452416" containerName="otc-container" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840070 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a66ef0-303f-452f-ade2-30c248452416" containerName="otc-container" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840081 4988 cpu_manager.go:414] "containerMap: removing container" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="kube-rbac-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840086 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="kube-rbac-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840093 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="prom-label-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840097 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="prom-label-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840116 4988 cpu_manager.go:414] "containerMap: removing container" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="prom-label-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840122 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="prom-label-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840130 4988 cpu_manager.go:414] "containerMap: removing container" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="init-config-reloader" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840135 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="init-config-reloader" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840143 4988 cpu_manager.go:414] "containerMap: removing container" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="method-gate" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840147 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="method-gate" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840154 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="alertmanager" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840158 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="alertmanager" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840209 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="prom-label-proxy" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840216 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="method-gate" Oct 30 21:18:28.840215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840222 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="alertmanager" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840230 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="kube-rbac-proxy" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840236 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="prometheus" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840242 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96216c3-f3c3-4293-a4db-f793381e1a51" containerName="config-reloader" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840246 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="config-reloader" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840250 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="deeeab09-eb4d-486c-8d3e-b9beca0b768b" containerName="thanos-sidecar" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840256 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="method-gate" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840261 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a66ef0-303f-452f-ade2-30c248452416" containerName="otc-container" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840267 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d80f20-1d3c-44bd-99c9-3d2ed1949f8d" containerName="kube-rbac-proxy" Oct 30 21:18:28.841034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.840272 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd1d521-f511-4f19-993d-38c4e069d692" containerName="prom-label-proxy" Oct 30 21:18:28.845122 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.845090 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:28.914275 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.914235 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.958826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.958788 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:18:28.959930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.959826 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:18:28.959930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.959831 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-x5mw4\"" Oct 30 21:18:28.959930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.959894 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:18:28.960147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.960051 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:28.960147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.960095 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:28.960147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.960122 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwvz\" (UniqueName: \"kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:28.960282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:28.960237 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.008854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.008812 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:18:29.008854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.008816 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:18:29.009058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.008865 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 21:18:29.061194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.061139 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.061722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.061229 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.061722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.061275 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.061722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.061312 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwvz\" (UniqueName: \"kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.061910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.061883 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.063893 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.063862 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.064002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.063903 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.123032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.122961 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwvz\" (UniqueName: \"kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz\") pod \"data-science-collector-collector-1\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.158352 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.158321 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:18:29.534235 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.534201 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:18:29.537795 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:18:29.537763 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cc2e34_ef3a_412a_b06f_32f6e5c1dfe0.slice/crio-44145d715143c4ddbced9176f1b42eee97fa4291f14615d6ef461f67695517b0 WatchSource:0}: Error finding container 44145d715143c4ddbced9176f1b42eee97fa4291f14615d6ef461f67695517b0: Status 404 returned error can't find the container with id 44145d715143c4ddbced9176f1b42eee97fa4291f14615d6ef461f67695517b0 Oct 30 21:18:29.539868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.539836 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:29.539937 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.539897 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:29.646762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.646726 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0","Type":"ContainerStarted","Data":"ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d"} Oct 30 21:18:29.646951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.646784 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0","Type":"ContainerStarted","Data":"44145d715143c4ddbced9176f1b42eee97fa4291f14615d6ef461f67695517b0"} Oct 30 21:18:29.765685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:29.765626 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.765610517 podStartE2EDuration="1.765610517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:29.764381354 +0000 UTC m=+2537.871877270" watchObservedRunningTime="2026-10-30 21:18:29.765610517 +0000 UTC m=+2537.873106429" Oct 30 21:18:30.374637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.374605 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:18:30.378567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.378400 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.415157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.415113 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:30.431438 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.431377 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:18:30.431438 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.431387 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:18:30.431651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.431386 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 21:18:30.431651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.431467 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 21:18:30.431651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.431515 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 21:18:30.431651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.431627 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-5st99\"" Oct 30 21:18:30.481034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.480995 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 21:18:30.481199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.481045 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 21:18:30.482466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.482441 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:18:30.575012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.574970 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575014 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575113 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575222 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575333 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575273 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575333 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575298 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575333 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575314 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564tx\" (UniqueName: \"kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.575487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.575403 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.632472 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.632433 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:30.676105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676061 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676114 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676145 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564tx\" (UniqueName: \"kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676181 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676209 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676237 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676276 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676306 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.676991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676958 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.677097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.676963 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.677097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.677065 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.677097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.677077 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.678839 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.678813 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.678976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.678913 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.679035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.678998 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:30.737965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:30.737925 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564tx\" (UniqueName: \"kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:32.897555 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:32.897508 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.002764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.002725 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.099008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.098963 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.099210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.099100 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.151380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.151293 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.288760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.288721 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.354552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.354524 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:33.699987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:33.699943 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:34.220238 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:18:34.220201 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6c62a0_4cc2_4118_b45d_99bdc60e335c.slice/crio-b4d5d88e004ae6ac51596a12d50974a756d0f50d6e2476d4500bab8ef37910e8 WatchSource:0}: Error finding container b4d5d88e004ae6ac51596a12d50974a756d0f50d6e2476d4500bab8ef37910e8: Status 404 returned error can't find the container with id b4d5d88e004ae6ac51596a12d50974a756d0f50d6e2476d4500bab8ef37910e8 Oct 30 21:18:34.254519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.254485 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:18:34.677449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.677380 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerStarted","Data":"fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f"} Oct 30 21:18:34.677660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.677476 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:34.677660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.677500 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerStarted","Data":"e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4"} Oct 30 21:18:34.677660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.677521 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerStarted","Data":"dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b"} Oct 30 21:18:34.677660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.677541 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerStarted","Data":"b4d5d88e004ae6ac51596a12d50974a756d0f50d6e2476d4500bab8ef37910e8"} Oct 30 21:18:34.677660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.677565 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:34.680254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.680226 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:34.686008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.685983 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:34.800959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:34.800904 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.800888191 podStartE2EDuration="4.800888191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:34.799764618 +0000 UTC m=+2542.907260528" watchObservedRunningTime="2026-10-30 21:18:34.800888191 +0000 UTC m=+2542.908384106" Oct 30 21:18:35.681993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:35.681960 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:18:56.394315 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:18:56.394230 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:18:56.688237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:18:56.688143 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:19:06.687581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:06.687535 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:19:09.266371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:09.266325 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:09.465108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:09.465067 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:16.097601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:16.097564 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:16.687772 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:16.687730 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:22.194336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.194298 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:22.194935 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.194765 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" containerName="otc-container" containerID="cri-o://ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d" gracePeriod=30 Oct 30 21:19:22.222506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.222470 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.421019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.420981 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.453542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.453475 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:22.634306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.634264 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs\") pod \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " Oct 30 21:19:22.634546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.634323 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs\") pod \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " Oct 30 21:19:22.634546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.634458 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz\" (UniqueName: \"kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz\") pod \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " Oct 30 21:19:22.634546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.634489 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal\") pod \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\" (UID: \"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0\") " Oct 30 21:19:22.634913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.634884 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal" pod "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" (UID: "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:22.636721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.636690 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs" pod "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" (UID: "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:22.636836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.636804 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs" pod "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" (UID: "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:22.636836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.636819 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz" pod "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" (UID: "18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0"). InnerVolumeSpecName "kube-api-access-xnwvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:22.637370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.637340 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.735303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.735211 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:22.735303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.735242 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:22.735303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.735260 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnwvz\" (UniqueName: \"kubernetes.io/projected/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-kube-api-access-xnwvz\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:22.735303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.735275 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:22.879393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.879354 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:22.879757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.879702 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo" containerID="cri-o://dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b" gracePeriod=30 Oct 30 21:19:22.880331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.880182 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway-opa" containerID="cri-o://fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f" gracePeriod=30 Oct 30 21:19:22.880440 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.880228 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway" containerID="cri-o://e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4" gracePeriod=30 Oct 30 21:19:22.895660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.895621 4988 generic.go:356] "Generic (PLEG): container finished" podID="18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" containerID="ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d" exitCode=0 Oct 30 21:19:22.895865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.895682 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:22.895865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.895705 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0","Type":"ContainerDied","Data":"ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d"} Oct 30 21:19:22.895865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.895763 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0","Type":"ContainerDied","Data":"44145d715143c4ddbced9176f1b42eee97fa4291f14615d6ef461f67695517b0"} Oct 30 21:19:22.895865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.895795 4988 scope.go:122] "RemoveContainer" containerID="ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d" Oct 30 21:19:22.941265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.941228 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.993822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.993801 4988 scope.go:122] "RemoveContainer" containerID="ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d" Oct 30 21:19:22.994248 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:22.994224 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d\": container with ID starting with ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d not found: ID does not exist" containerID="ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d" Oct 30 21:19:22.994310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:22.994263 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d"} err="failed to get container status \"ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d\": rpc error: code = NotFound desc = could not find container \"ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d\": container with ID starting with ae0b129ad435561b1b373dd54feaf24f49271a346a95dbd3560b8ca61265640d not found: ID does not exist" Oct 30 21:19:23.113123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.113086 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:23.118559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.118526 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:23.177348 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.177313 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:23.328541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.328451 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:23.902669 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.902634 4988 generic.go:356] "Generic (PLEG): container finished" podID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerID="fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f" exitCode=0 Oct 30 21:19:23.902669 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.902664 4988 generic.go:356] "Generic (PLEG): container finished" podID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerID="dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b" exitCode=0 Oct 30 21:19:23.902929 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.902708 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerDied","Data":"fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f"} Oct 30 21:19:23.902929 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:23.902759 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerDied","Data":"dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b"} Oct 30 21:19:24.015897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.015865 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" path="/var/lib/kubelet/pods/18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0/volumes" Oct 30 21:19:24.142368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.142340 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:24.249618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249528 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx\" (UniqueName: \"kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249577 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249618 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249657 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249677 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249724 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249760 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249805 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.249969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.249895 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\" (UID: \"cd6c62a0-4cc2-4118-b45d-99bdc60e335c\") " Oct 30 21:19:24.250267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.250043 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:24.250267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.250173 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.250373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.250357 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:24.250534 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.250502 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:24.250675 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.250568 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:24.252165 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.252120 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "kube-api-access-564tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:24.252294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.252262 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:24.252358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.252336 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:24.252358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.252348 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:24.253153 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.253135 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "cd6c62a0-4cc2-4118-b45d-99bdc60e335c" (UID: "cd6c62a0-4cc2-4118-b45d-99bdc60e335c"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:19:24.351133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351094 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.351133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351127 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.351133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351139 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.351639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351168 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:19:24.351639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351178 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564tx\" (UniqueName: \"kubernetes.io/projected/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-kube-api-access-564tx\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.351639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351188 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.351639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351199 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.351639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.351208 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cd6c62a0-4cc2-4118-b45d-99bdc60e335c-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:24.359991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.359967 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:19:24.452226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.452185 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 21:19:24.909805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.909770 4988 generic.go:356] "Generic (PLEG): container finished" podID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerID="e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4" exitCode=0 Oct 30 21:19:24.910027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.909828 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:24.910027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.909852 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerDied","Data":"e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4"} Oct 30 21:19:24.910027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.909921 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cd6c62a0-4cc2-4118-b45d-99bdc60e335c","Type":"ContainerDied","Data":"b4d5d88e004ae6ac51596a12d50974a756d0f50d6e2476d4500bab8ef37910e8"} Oct 30 21:19:24.910027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.909955 4988 scope.go:122] "RemoveContainer" containerID="fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f" Oct 30 21:19:24.919038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.919014 4988 scope.go:122] "RemoveContainer" containerID="e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4" Oct 30 21:19:24.926850 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.926828 4988 scope.go:122] "RemoveContainer" containerID="dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b" Oct 30 21:19:24.934780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.934755 4988 scope.go:122] "RemoveContainer" containerID="fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f" Oct 30 21:19:24.935122 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:24.935102 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f\": container with ID starting with fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f not found: ID does not exist" containerID="fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f" Oct 30 21:19:24.935173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.935132 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f"} err="failed to get container status \"fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f\": rpc error: code = NotFound desc = could not find container \"fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f\": container with ID starting with fd73638570a5c10fc758e4e31406bb1dd1a0222e6d0978a3824f7d067440416f not found: ID does not exist" Oct 30 21:19:24.935173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.935151 4988 scope.go:122] "RemoveContainer" containerID="e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4" Oct 30 21:19:24.935431 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:24.935389 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4\": container with ID starting with e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4 not found: ID does not exist" containerID="e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4" Oct 30 21:19:24.935574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.935545 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4"} err="failed to get container status \"e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4\": rpc error: code = NotFound desc = could not find container \"e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4\": container with ID starting with e072f32de2409fdb876e9502b2d8f1d944ed13e54c39886cf1ec78d9ba6221c4 not found: ID does not exist" Oct 30 21:19:24.935574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.935572 4988 scope.go:122] "RemoveContainer" containerID="dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b" Oct 30 21:19:24.935832 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:24.935815 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b\": container with ID starting with dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b not found: ID does not exist" containerID="dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b" Oct 30 21:19:24.935876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:24.935837 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b"} err="failed to get container status \"dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b\": rpc error: code = NotFound desc = could not find container \"dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b\": container with ID starting with dc7d6e9197aec23aef4da317929e7428ee3823ebac1ecc5f31a41ce6c2c9148b not found: ID does not exist" Oct 30 21:19:25.075081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:25.075047 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:25.125260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:25.125218 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:26.015752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:26.015716 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" path="/var/lib/kubelet/pods/cd6c62a0-4cc2-4118-b45d-99bdc60e335c/volumes" Oct 30 21:19:26.395524 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:26.395485 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:19:28.477788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:28.477750 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:28.678710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:28.678671 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:28.821266 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:28.821165 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:28.969346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:28.969304 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.602104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.602067 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.622505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622468 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:30.622790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622778 4988 cpu_manager.go:414] "containerMap: removing container" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway" Oct 30 21:19:30.622837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622792 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway" Oct 30 21:19:30.622837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622805 4988 cpu_manager.go:414] "containerMap: removing container" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo" Oct 30 21:19:30.622837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622810 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo" Oct 30 21:19:30.622837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622821 4988 cpu_manager.go:414] "containerMap: removing container" podUID="18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" containerName="otc-container" Oct 30 21:19:30.622837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622826 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" containerName="otc-container" Oct 30 21:19:30.622981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622848 4988 cpu_manager.go:414] "containerMap: removing container" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway-opa" Oct 30 21:19:30.622981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622853 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway-opa" Oct 30 21:19:30.622981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622901 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway-opa" Oct 30 21:19:30.622981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622910 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo-gateway" Oct 30 21:19:30.622981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622917 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6c62a0-4cc2-4118-b45d-99bdc60e335c" containerName="tempo" Oct 30 21:19:30.622981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.622923 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cc2e34-ef3a-412a-b06f-32f6e5c1dfe0" containerName="otc-container" Oct 30 21:19:30.627674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.627649 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.686939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.686904 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:30.729977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.729939 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-z8w5p\"" Oct 30 21:19:30.730168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.729956 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:19:30.730168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.729956 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:19:30.730168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.730004 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:19:30.733556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.733530 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:19:30.733667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.733601 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 21:19:30.751916 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.751878 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.803442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.802683 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.803442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.802819 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.803442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.802866 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpm7r\" (UniqueName: \"kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.803442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.802999 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.897855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.897815 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.904365 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.904333 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.904558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.904374 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpm7r\" (UniqueName: \"kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.904558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.904403 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.904558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.904467 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.905207 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.905178 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.906834 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.906811 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.906915 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.906871 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:30.966614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:30.966582 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpm7r\" (UniqueName: \"kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r\") pod \"data-science-collector-collector-1\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:31.240341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:31.240258 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:31.648818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:31.648777 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:31.652363 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:19:31.652324 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459921ee_8605_4f08_8ddf_7964e773e4d0.slice/crio-0d33cc5762bbbaf04c225eeb9362d52016531164f6e84bc5d267b8a5fd5b31e0 WatchSource:0}: Error finding container 0d33cc5762bbbaf04c225eeb9362d52016531164f6e84bc5d267b8a5fd5b31e0: Status 404 returned error can't find the container with id 0d33cc5762bbbaf04c225eeb9362d52016531164f6e84bc5d267b8a5fd5b31e0 Oct 30 21:19:31.654322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:31.654289 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:31.654444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:31.654361 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:19:31.942931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:31.942846 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"459921ee-8605-4f08-8ddf-7964e773e4d0","Type":"ContainerStarted","Data":"f1ebc63c2207a17cc39a5223f935b61c854855c8a179aab8eaf886aa9f02d4c4"} Oct 30 21:19:31.942931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:31.942907 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"459921ee-8605-4f08-8ddf-7964e773e4d0","Type":"ContainerStarted","Data":"0d33cc5762bbbaf04c225eeb9362d52016531164f6e84bc5d267b8a5fd5b31e0"} Oct 30 21:19:32.060939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.060884 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.06086778 podStartE2EDuration="2.06086778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:32.059989443 +0000 UTC m=+2600.167485355" watchObservedRunningTime="2026-10-30 21:19:32.06086778 +0000 UTC m=+2600.168363691" Oct 30 21:19:32.636256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.636216 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:32.640483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.640458 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.676899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.676866 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:32.696167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.696125 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 21:19:32.696167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.696125 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:19:32.696627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.696609 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-m4pkp\"" Oct 30 21:19:32.696739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.696713 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 21:19:32.696785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.696738 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 21:19:32.745115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.745074 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 21:19:32.745115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.745074 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:19:32.745585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.745563 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 21:19:32.746734 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.746715 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:32.877378 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.877336 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:32.921300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921205 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921250 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4h75\" (UniqueName: \"kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921515 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921379 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921515 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921448 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921536 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921562 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921649 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921591 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:32.921683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:32.921648 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022211 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022268 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022310 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022339 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022378 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022403 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4h75\" (UniqueName: \"kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022464 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.022748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.022666 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.023229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.023197 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.023362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.023225 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.023457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.023387 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.023457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.023439 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.024952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.024924 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.025073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.025012 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.025139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.025093 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.084367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.084332 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4h75\" (UniqueName: \"kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.223847 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.223760 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.328605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.328557 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.425802 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.425757 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.425981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.425907 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.478236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.478157 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.617634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.617585 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.682884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.682852 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:33.857633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:33.857548 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:34.340560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.340526 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:34.344216 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:19:34.344186 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431935f7_065b_4e93_9d18_120cb9c24b77.slice/crio-97011bc5b4ad5f53dce7e51297b52bc753b2d43f3e560a3edbe5e9da1d6c8f29 WatchSource:0}: Error finding container 97011bc5b4ad5f53dce7e51297b52bc753b2d43f3e560a3edbe5e9da1d6c8f29: Status 404 returned error can't find the container with id 97011bc5b4ad5f53dce7e51297b52bc753b2d43f3e560a3edbe5e9da1d6c8f29 Oct 30 21:19:34.958248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958210 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerStarted","Data":"d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021"} Oct 30 21:19:34.958754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958309 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerStarted","Data":"6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f"} Oct 30 21:19:34.958754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958342 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerStarted","Data":"81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c"} Oct 30 21:19:34.958754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958366 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerStarted","Data":"97011bc5b4ad5f53dce7e51297b52bc753b2d43f3e560a3edbe5e9da1d6c8f29"} Oct 30 21:19:34.958754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958395 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:34.958754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958435 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:34.958754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.958455 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:34.961033 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.961004 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:34.967050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:34.967023 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:35.081843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:35.081784 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=3.081772115 podStartE2EDuration="3.081772115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:35.081126356 +0000 UTC m=+2603.188622272" watchObservedRunningTime="2026-10-30 21:19:35.081772115 +0000 UTC m=+2603.189268027" Oct 30 21:19:45.520718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:45.520671 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:52.828085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:52.828047 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:52.828649 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:52.828266 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="459921ee-8605-4f08-8ddf-7964e773e4d0" containerName="otc-container" containerID="cri-o://f1ebc63c2207a17cc39a5223f935b61c854855c8a179aab8eaf886aa9f02d4c4" gracePeriod=30 Oct 30 21:19:52.848799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:52.848764 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.001152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.001112 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.038265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.038233 4988 generic.go:356] "Generic (PLEG): container finished" podID="459921ee-8605-4f08-8ddf-7964e773e4d0" containerID="f1ebc63c2207a17cc39a5223f935b61c854855c8a179aab8eaf886aa9f02d4c4" exitCode=0 Oct 30 21:19:53.038513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.038303 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"459921ee-8605-4f08-8ddf-7964e773e4d0","Type":"ContainerDied","Data":"f1ebc63c2207a17cc39a5223f935b61c854855c8a179aab8eaf886aa9f02d4c4"} Oct 30 21:19:53.096510 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.096479 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:53.148080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.148047 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.185646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.185607 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal\") pod \"459921ee-8605-4f08-8ddf-7964e773e4d0\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " Oct 30 21:19:53.185845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.185671 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs\") pod \"459921ee-8605-4f08-8ddf-7964e773e4d0\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " Oct 30 21:19:53.185845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.185719 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r\" (UniqueName: \"kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r\") pod \"459921ee-8605-4f08-8ddf-7964e773e4d0\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " Oct 30 21:19:53.185845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.185781 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs\") pod \"459921ee-8605-4f08-8ddf-7964e773e4d0\" (UID: \"459921ee-8605-4f08-8ddf-7964e773e4d0\") " Oct 30 21:19:53.186074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.186052 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal" pod "459921ee-8605-4f08-8ddf-7964e773e4d0" (UID: "459921ee-8605-4f08-8ddf-7964e773e4d0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:53.187913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.187881 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs" pod "459921ee-8605-4f08-8ddf-7964e773e4d0" (UID: "459921ee-8605-4f08-8ddf-7964e773e4d0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:53.188019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.187910 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs" pod "459921ee-8605-4f08-8ddf-7964e773e4d0" (UID: "459921ee-8605-4f08-8ddf-7964e773e4d0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:53.188019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.187974 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r" pod "459921ee-8605-4f08-8ddf-7964e773e4d0" (UID: "459921ee-8605-4f08-8ddf-7964e773e4d0"). InnerVolumeSpecName "kube-api-access-tpm7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:53.286748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.286711 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:53.286748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.286745 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/459921ee-8605-4f08-8ddf-7964e773e4d0-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:53.286748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.286755 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/459921ee-8605-4f08-8ddf-7964e773e4d0-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:53.286999 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.286764 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpm7r\" (UniqueName: \"kubernetes.io/projected/459921ee-8605-4f08-8ddf-7964e773e4d0-kube-api-access-tpm7r\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:53.505817 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.505782 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:53.506023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.505994 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo" containerID="cri-o://81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c" gracePeriod=30 Oct 30 21:19:53.506171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.506061 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway-opa" containerID="cri-o://d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021" gracePeriod=30 Oct 30 21:19:53.506171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.506096 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway" containerID="cri-o://6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f" gracePeriod=30 Oct 30 21:19:53.510172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.508201 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:19:53.555192 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.555157 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.803941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.803835 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:53.951736 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:53.951698 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:54.045394 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.045358 4988 generic.go:356] "Generic (PLEG): container finished" podID="431935f7-065b-4e93-9d18-120cb9c24b77" containerID="d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021" exitCode=0 Oct 30 21:19:54.045394 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.045388 4988 generic.go:356] "Generic (PLEG): container finished" podID="431935f7-065b-4e93-9d18-120cb9c24b77" containerID="81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c" exitCode=0 Oct 30 21:19:54.045647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.045446 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerDied","Data":"d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021"} Oct 30 21:19:54.045647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.045500 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerDied","Data":"81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c"} Oct 30 21:19:54.046932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.046901 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"459921ee-8605-4f08-8ddf-7964e773e4d0","Type":"ContainerDied","Data":"0d33cc5762bbbaf04c225eeb9362d52016531164f6e84bc5d267b8a5fd5b31e0"} Oct 30 21:19:54.046932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.046930 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:19:54.047145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.046966 4988 scope.go:122] "RemoveContainer" containerID="f1ebc63c2207a17cc39a5223f935b61c854855c8a179aab8eaf886aa9f02d4c4" Oct 30 21:19:54.211635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.211595 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:54.215700 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.215673 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:19:54.750401 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.750373 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:54.799854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.799808 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.799863 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75\" (UniqueName: \"kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.799907 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.799936 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.799965 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800003 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800117 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800148 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800235 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"431935f7-065b-4e93-9d18-120cb9c24b77\" (UID: \"431935f7-065b-4e93-9d18-120cb9c24b77\") " Oct 30 21:19:54.800391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800293 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:54.800620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800467 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:54.800620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800496 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:54.800620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800512 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.800837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.800800 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:54.802507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.802482 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "kube-api-access-j4h75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:54.802639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.802606 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:54.802753 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.802723 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:54.803036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.803012 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:54.803101 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.803016 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "431935f7-065b-4e93-9d18-120cb9c24b77" (UID: "431935f7-065b-4e93-9d18-120cb9c24b77"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:19:54.901618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901580 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.901618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901615 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4h75\" (UniqueName: \"kubernetes.io/projected/431935f7-065b-4e93-9d18-120cb9c24b77-kube-api-access-j4h75\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.901884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901629 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.901884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901644 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.901884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901659 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.901884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901671 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.901884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901707 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:19:54.901884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.901722 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/431935f7-065b-4e93-9d18-120cb9c24b77-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:54.911027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:54.911002 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:19:55.002584 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.002542 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 21:19:55.054050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.054014 4988 generic.go:356] "Generic (PLEG): container finished" podID="431935f7-065b-4e93-9d18-120cb9c24b77" containerID="6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f" exitCode=0 Oct 30 21:19:55.054228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.054082 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerDied","Data":"6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f"} Oct 30 21:19:55.054228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.054084 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:19:55.054228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.054127 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"431935f7-065b-4e93-9d18-120cb9c24b77","Type":"ContainerDied","Data":"97011bc5b4ad5f53dce7e51297b52bc753b2d43f3e560a3edbe5e9da1d6c8f29"} Oct 30 21:19:55.054228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.054149 4988 scope.go:122] "RemoveContainer" containerID="d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021" Oct 30 21:19:55.062739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.062715 4988 scope.go:122] "RemoveContainer" containerID="6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f" Oct 30 21:19:55.071222 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.071202 4988 scope.go:122] "RemoveContainer" containerID="81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c" Oct 30 21:19:55.079157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.079133 4988 scope.go:122] "RemoveContainer" containerID="d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021" Oct 30 21:19:55.079508 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:55.079489 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021\": container with ID starting with d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021 not found: ID does not exist" containerID="d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021" Oct 30 21:19:55.079604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.079521 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021"} err="failed to get container status \"d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021\": rpc error: code = NotFound desc = could not find container \"d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021\": container with ID starting with d029fa58e6d1f9ce303a791ad3d39a6e6559ffca08a0e5140e95282da4ee8021 not found: ID does not exist" Oct 30 21:19:55.079604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.079547 4988 scope.go:122] "RemoveContainer" containerID="6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f" Oct 30 21:19:55.079871 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:55.079851 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f\": container with ID starting with 6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f not found: ID does not exist" containerID="6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f" Oct 30 21:19:55.080116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.079877 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f"} err="failed to get container status \"6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f\": rpc error: code = NotFound desc = could not find container \"6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f\": container with ID starting with 6e4ddd3c3298dbea0ed026ee9dd91b9fc0aa61fcf2e01e02d222c456aba4c88f not found: ID does not exist" Oct 30 21:19:55.080116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.079892 4988 scope.go:122] "RemoveContainer" containerID="81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c" Oct 30 21:19:55.080116 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:55.080101 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c\": container with ID starting with 81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c not found: ID does not exist" containerID="81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c" Oct 30 21:19:55.080271 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.080119 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c"} err="failed to get container status \"81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c\": rpc error: code = NotFound desc = could not find container \"81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c\": container with ID starting with 81a8480f7b6b327682a905e19ccac471955532806581eaa696cc1a31c8effb9c not found: ID does not exist" Oct 30 21:19:55.218405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.218323 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:55.224602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:55.224567 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:19:56.015783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:56.015740 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" path="/var/lib/kubelet/pods/431935f7-065b-4e93-9d18-120cb9c24b77/volumes" Oct 30 21:19:56.016435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:56.016393 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459921ee-8605-4f08-8ddf-7964e773e4d0" path="/var/lib/kubelet/pods/459921ee-8605-4f08-8ddf-7964e773e4d0/volumes" Oct 30 21:19:56.394783 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:19:56.394750 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:19:59.098574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:59.098529 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:59.299253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:59.299210 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:59.445483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:19:59.445444 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:01.222921 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.222882 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:01.263545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263508 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:20:01.263850 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263837 4988 cpu_manager.go:414] "containerMap: removing container" podUID="459921ee-8605-4f08-8ddf-7964e773e4d0" containerName="otc-container" Oct 30 21:20:01.263897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263852 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459921ee-8605-4f08-8ddf-7964e773e4d0" containerName="otc-container" Oct 30 21:20:01.263897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263869 4988 cpu_manager.go:414] "containerMap: removing container" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway-opa" Oct 30 21:20:01.263897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263874 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway-opa" Oct 30 21:20:01.263897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263891 4988 cpu_manager.go:414] "containerMap: removing container" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway" Oct 30 21:20:01.263897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263896 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway" Oct 30 21:20:01.264056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263904 4988 cpu_manager.go:414] "containerMap: removing container" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo" Oct 30 21:20:01.264056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263909 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo" Oct 30 21:20:01.264056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263968 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="459921ee-8605-4f08-8ddf-7964e773e4d0" containerName="otc-container" Oct 30 21:20:01.264056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263976 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway-opa" Oct 30 21:20:01.264056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263983 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo-gateway" Oct 30 21:20:01.264056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.263988 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="431935f7-065b-4e93-9d18-120cb9c24b77" containerName="tempo" Oct 30 21:20:01.268758 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.268733 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.326856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.326815 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:20:01.353347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.353310 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghhw\" (UniqueName: \"kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.353347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.353357 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.353609 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.353385 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.353609 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.353531 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.371984 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.371951 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:01.374256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.374235 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:20:01.374483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.374468 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:20:01.374555 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.374539 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:20:01.374609 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.374562 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 21:20:01.374663 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.374652 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-fbjqp\"" Oct 30 21:20:01.374714 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.374697 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:20:01.454099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.454066 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghhw\" (UniqueName: \"kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.454099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.454106 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.454364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.454130 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.454364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.454148 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.454997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.454975 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.456727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.456699 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.456833 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.456725 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.515352 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.515270 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghhw\" (UniqueName: \"kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw\") pod \"data-science-collector-collector-1\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.519559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.519526 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:01.582549 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.582506 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:01.960918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.960885 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:20:01.964291 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:20:01.964259 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode550ef8c_7a72_460b_aede_c872dda15a05.slice/crio-1ea83c6fb5ae04c6b9aeba9bbad1ae408fcaf3b71bbb2f0323e0b0afe95c8d2e WatchSource:0}: Error finding container 1ea83c6fb5ae04c6b9aeba9bbad1ae408fcaf3b71bbb2f0323e0b0afe95c8d2e: Status 404 returned error can't find the container with id 1ea83c6fb5ae04c6b9aeba9bbad1ae408fcaf3b71bbb2f0323e0b0afe95c8d2e Oct 30 21:20:01.966402 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.966365 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:01.966529 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:01.966462 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:02.088910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:02.088859 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e550ef8c-7a72-460b-aede-c872dda15a05","Type":"ContainerStarted","Data":"6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c"} Oct 30 21:20:02.089072 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:02.088938 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e550ef8c-7a72-460b-aede-c872dda15a05","Type":"ContainerStarted","Data":"1ea83c6fb5ae04c6b9aeba9bbad1ae408fcaf3b71bbb2f0323e0b0afe95c8d2e"} Oct 30 21:20:02.207376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:02.207304 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.207287644 podStartE2EDuration="1.207287644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:02.206235185 +0000 UTC m=+2630.313731101" watchObservedRunningTime="2026-10-30 21:20:02.207287644 +0000 UTC m=+2630.314783556" Oct 30 21:20:03.124656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.124618 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:20:03.128327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.128304 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.164905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.164870 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:03.180648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.180609 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 21:20:03.180648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.180631 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:20:03.180648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.180628 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:20:03.180897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.180627 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 21:20:03.230079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.230041 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 21:20:03.230079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.230049 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-kplpw\"" Oct 30 21:20:03.230336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.230140 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 21:20:03.230336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.230223 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 21:20:03.234257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.234228 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:20:03.356082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.356048 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:03.372156 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372118 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372161 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt62m\" (UniqueName: \"kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372231 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372278 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372337 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372372 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372489 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.372542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.372525 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473240 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473203 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473240 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473243 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt62m\" (UniqueName: \"kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473263 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473291 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473327 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473608 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473667 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.473740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.473705 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.474151 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.474117 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.474291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.474184 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.474368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.474329 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.474437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.474373 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.476721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.476696 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.476721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.476704 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.476721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.476697 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.541984 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.541953 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt62m\" (UniqueName: \"kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:03.977825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:03.977779 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.081966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.081936 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.179700 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.179658 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.180120 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.179881 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.231735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.231644 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.368849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.368814 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.434488 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.434457 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:04.645024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:04.644979 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:05.131846 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:20:05.131811 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532c6f1e_ff4e_4683_aa6f_6c0da4455088.slice/crio-ea6155a83eb530ca80fdb73cac32b70d7e281dc3779f728525ab29c4b2c8b16b WatchSource:0}: Error finding container ea6155a83eb530ca80fdb73cac32b70d7e281dc3779f728525ab29c4b2c8b16b: Status 404 returned error can't find the container with id ea6155a83eb530ca80fdb73cac32b70d7e281dc3779f728525ab29c4b2c8b16b Oct 30 21:20:05.178045 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:05.178008 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:20:06.109465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.109422 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerStarted","Data":"e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a"} Oct 30 21:20:06.110003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.109490 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerStarted","Data":"32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73"} Oct 30 21:20:06.110003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.109511 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerStarted","Data":"8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f"} Oct 30 21:20:06.110003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.109537 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:06.110003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.109565 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerStarted","Data":"ea6155a83eb530ca80fdb73cac32b70d7e281dc3779f728525ab29c4b2c8b16b"} Oct 30 21:20:06.110003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.109590 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:06.112317 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.112288 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:06.119532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.119505 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:06.338144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:06.338090 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=3.338074919 podStartE2EDuration="3.338074919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:06.28833144 +0000 UTC m=+2634.395827351" watchObservedRunningTime="2026-10-30 21:20:06.338074919 +0000 UTC m=+2634.445570831" Oct 30 21:20:07.115029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:07.114995 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:16.450228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:16.450188 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:26.395261 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:20:26.395175 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:20:28.122024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:28.121971 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:20:38.121790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:38.121742 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:20:48.121883 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:48.121840 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:52.698525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:52.698489 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:20:52.699067 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:52.698691 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e550ef8c-7a72-460b-aede-c872dda15a05" containerName="otc-container" containerID="cri-o://6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c" gracePeriod=30 Oct 30 21:20:52.733729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:52.733682 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.878014 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:52.877975 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:52.953117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:52.953052 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:53.088635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.088587 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs\") pod \"e550ef8c-7a72-460b-aede-c872dda15a05\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " Oct 30 21:20:53.088811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.088675 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal\") pod \"e550ef8c-7a72-460b-aede-c872dda15a05\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " Oct 30 21:20:53.088811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.088715 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw\" (UniqueName: \"kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw\") pod \"e550ef8c-7a72-460b-aede-c872dda15a05\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " Oct 30 21:20:53.088939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.088861 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs\") pod \"e550ef8c-7a72-460b-aede-c872dda15a05\" (UID: \"e550ef8c-7a72-460b-aede-c872dda15a05\") " Oct 30 21:20:53.089095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.089068 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal" pod "e550ef8c-7a72-460b-aede-c872dda15a05" (UID: "e550ef8c-7a72-460b-aede-c872dda15a05"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:53.091034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.090999 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw" pod "e550ef8c-7a72-460b-aede-c872dda15a05" (UID: "e550ef8c-7a72-460b-aede-c872dda15a05"). InnerVolumeSpecName "kube-api-access-nghhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:53.091179 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.091136 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs" pod "e550ef8c-7a72-460b-aede-c872dda15a05" (UID: "e550ef8c-7a72-460b-aede-c872dda15a05"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:53.091349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.091321 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs" pod "e550ef8c-7a72-460b-aede-c872dda15a05" (UID: "e550ef8c-7a72-460b-aede-c872dda15a05"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:53.122264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.122223 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:53.189806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.189770 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:53.189998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.189940 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e550ef8c-7a72-460b-aede-c872dda15a05-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:53.189998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.189967 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e550ef8c-7a72-460b-aede-c872dda15a05-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:53.189998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.189978 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghhw\" (UniqueName: \"kubernetes.io/projected/e550ef8c-7a72-460b-aede-c872dda15a05-kube-api-access-nghhw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:53.324514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.324390 4988 generic.go:356] "Generic (PLEG): container finished" podID="e550ef8c-7a72-460b-aede-c872dda15a05" containerID="6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c" exitCode=0 Oct 30 21:20:53.324514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.324470 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:20:53.324514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.324494 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e550ef8c-7a72-460b-aede-c872dda15a05","Type":"ContainerDied","Data":"6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c"} Oct 30 21:20:53.324756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.324550 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e550ef8c-7a72-460b-aede-c872dda15a05","Type":"ContainerDied","Data":"1ea83c6fb5ae04c6b9aeba9bbad1ae408fcaf3b71bbb2f0323e0b0afe95c8d2e"} Oct 30 21:20:53.324756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.324571 4988 scope.go:122] "RemoveContainer" containerID="6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c" Oct 30 21:20:53.334108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.334088 4988 scope.go:122] "RemoveContainer" containerID="6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c" Oct 30 21:20:53.334453 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:20:53.334430 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c\": container with ID starting with 6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c not found: ID does not exist" containerID="6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c" Oct 30 21:20:53.334513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.334464 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c"} err="failed to get container status \"6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c\": rpc error: code = NotFound desc = could not find container \"6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c\": container with ID starting with 6288576fe101df3c08bd1d7f75e599ab7669208613f204b49684a98877eaf48c not found: ID does not exist" Oct 30 21:20:53.421780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.421748 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:20:53.422022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.421975 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo" containerID="cri-o://8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f" gracePeriod=30 Oct 30 21:20:53.422081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.422012 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway-opa" containerID="cri-o://e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a" gracePeriod=30 Oct 30 21:20:53.422135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.421996 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway" containerID="cri-o://32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73" gracePeriod=30 Oct 30 21:20:53.436866 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.436838 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:53.523166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.523131 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:20:53.526946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.526911 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:20:53.676585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.676539 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:53.820247 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:53.820214 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:54.015917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.015886 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e550ef8c-7a72-460b-aede-c872dda15a05" path="/var/lib/kubelet/pods/e550ef8c-7a72-460b-aede-c872dda15a05/volumes" Oct 30 21:20:54.332487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.332380 4988 generic.go:356] "Generic (PLEG): container finished" podID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerID="e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a" exitCode=0 Oct 30 21:20:54.332487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.332430 4988 generic.go:356] "Generic (PLEG): container finished" podID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerID="8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f" exitCode=0 Oct 30 21:20:54.332725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.332560 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerDied","Data":"e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a"} Oct 30 21:20:54.332725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.332664 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerDied","Data":"8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f"} Oct 30 21:20:54.681788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.681759 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:54.804815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.804775 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.804815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.804817 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.804870 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.804917 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m\" (UniqueName: \"kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.804947 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.804981 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805014 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805122 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805152 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls\") pod \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\" (UID: \"532c6f1e-ff4e-4683-aa6f-6c0da4455088\") " Oct 30 21:20:54.805645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805521 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:54.805645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805534 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:54.805645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805550 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:54.805813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.805655 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:54.807481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.807446 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "kube-api-access-jt62m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:54.807705 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.807678 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:54.807803 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.807706 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:54.807803 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.807731 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:54.808543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.808522 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "532c6f1e-ff4e-4683-aa6f-6c0da4455088" (UID: "532c6f1e-ff4e-4683-aa6f-6c0da4455088"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:20:54.906073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906037 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt62m\" (UniqueName: \"kubernetes.io/projected/532c6f1e-ff4e-4683-aa6f-6c0da4455088-kube-api-access-jt62m\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906069 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906080 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906100 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906129 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:20:54.906701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906140 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906152 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/532c6f1e-ff4e-4683-aa6f-6c0da4455088-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906162 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.906701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.906172 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/532c6f1e-ff4e-4683-aa6f-6c0da4455088-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:54.915851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:54.915828 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:20:55.007492 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.007450 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 21:20:55.338865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.338765 4988 generic.go:356] "Generic (PLEG): container finished" podID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerID="32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73" exitCode=0 Oct 30 21:20:55.338865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.338824 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:20:55.338865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.338818 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerDied","Data":"32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73"} Oct 30 21:20:55.339186 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.338897 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"532c6f1e-ff4e-4683-aa6f-6c0da4455088","Type":"ContainerDied","Data":"ea6155a83eb530ca80fdb73cac32b70d7e281dc3779f728525ab29c4b2c8b16b"} Oct 30 21:20:55.339186 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.338931 4988 scope.go:122] "RemoveContainer" containerID="e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a" Oct 30 21:20:55.347899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.347881 4988 scope.go:122] "RemoveContainer" containerID="32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73" Oct 30 21:20:55.355858 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.355832 4988 scope.go:122] "RemoveContainer" containerID="8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f" Oct 30 21:20:55.364495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.364472 4988 scope.go:122] "RemoveContainer" containerID="e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a" Oct 30 21:20:55.364852 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:20:55.364831 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a\": container with ID starting with e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a not found: ID does not exist" containerID="e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a" Oct 30 21:20:55.364948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.364864 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a"} err="failed to get container status \"e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a\": rpc error: code = NotFound desc = could not find container \"e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a\": container with ID starting with e8c03e1ec88b720968918e7aca40c850a184cb040488d2843e91c869de34d55a not found: ID does not exist" Oct 30 21:20:55.364948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.364889 4988 scope.go:122] "RemoveContainer" containerID="32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73" Oct 30 21:20:55.365197 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:20:55.365177 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73\": container with ID starting with 32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73 not found: ID does not exist" containerID="32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73" Oct 30 21:20:55.365243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.365210 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73"} err="failed to get container status \"32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73\": rpc error: code = NotFound desc = could not find container \"32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73\": container with ID starting with 32793bfa2b21af404fb648c154b974f1f12043154b09214cfcd2e6c1d916be73 not found: ID does not exist" Oct 30 21:20:55.365243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.365227 4988 scope.go:122] "RemoveContainer" containerID="8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f" Oct 30 21:20:55.365459 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:20:55.365437 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f\": container with ID starting with 8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f not found: ID does not exist" containerID="8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f" Oct 30 21:20:55.365516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.365467 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f"} err="failed to get container status \"8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f\": rpc error: code = NotFound desc = could not find container \"8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f\": container with ID starting with 8642583d8481584bfe1144145785bb9f5b4f100ea54fe014f21c9fdbc39c0a7f not found: ID does not exist" Oct 30 21:20:55.502988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.502948 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:20:55.552995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:55.552960 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:20:56.016136 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:20:56.016101 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" path="/var/lib/kubelet/pods/532c6f1e-ff4e-4683-aa6f-6c0da4455088/volumes" Oct 30 21:20:56.395882 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:20:56.395845 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:21:00.608950 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:00.608911 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:12.276886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:12.276778 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:21:12.288373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:12.288346 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:21:16.799243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:16.799206 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:26.395973 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:21:26.395934 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:21:31.263270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:31.263234 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:31.410082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:31.410036 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:31.564787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:31.564692 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:31.783092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:31.783054 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:33.589012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.588970 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:33.734803 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.734762 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:33.812722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.812679 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813622 4988 cpu_manager.go:414] "containerMap: removing container" podUID="e550ef8c-7a72-460b-aede-c872dda15a05" containerName="otc-container" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813650 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e550ef8c-7a72-460b-aede-c872dda15a05" containerName="otc-container" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813688 4988 cpu_manager.go:414] "containerMap: removing container" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813696 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813711 4988 cpu_manager.go:414] "containerMap: removing container" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway-opa" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813718 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway-opa" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813730 4988 cpu_manager.go:414] "containerMap: removing container" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813737 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813886 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e550ef8c-7a72-460b-aede-c872dda15a05" containerName="otc-container" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813906 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813915 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway-opa" Oct 30 21:21:33.814135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.813933 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="532c6f1e-ff4e-4683-aa6f-6c0da4455088" containerName="tempo-gateway" Oct 30 21:21:33.819178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.819141 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:33.888825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.888781 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:33.921757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.921715 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-phj8t\"" Oct 30 21:21:33.921757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.921722 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:21:33.922005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.921769 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:21:33.922894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.922865 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:21:33.923244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.923222 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:33.923303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.923286 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59fj\" (UniqueName: \"kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:33.923389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.923370 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:33.923540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.923516 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:33.971392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.971353 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:21:33.971392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.971368 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:21:33.971685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:33.971464 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 21:21:34.024521 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.024473 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59fj\" (UniqueName: \"kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.024521 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.024526 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.024765 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.024582 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.024765 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.024622 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.025220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.025198 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.027202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.027178 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.027322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.027232 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.038614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.038575 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:34.086165 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.086126 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59fj\" (UniqueName: \"kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj\") pod \"data-science-collector-collector-1\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.135968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.135927 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:21:34.515615 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.515507 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:21:34.547348 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:21:34.547312 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc502aa51_5e81_4c4f_b9a3_3fd5a753e2ef.slice/crio-0d344d8dcbec9878e93df88198b2c30c616b04533d797010104b0493681a687b WatchSource:0}: Error finding container 0d344d8dcbec9878e93df88198b2c30c616b04533d797010104b0493681a687b: Status 404 returned error can't find the container with id 0d344d8dcbec9878e93df88198b2c30c616b04533d797010104b0493681a687b Oct 30 21:21:34.549389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.549355 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:34.549516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:34.549447 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:21:35.523390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:35.523353 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef","Type":"ContainerStarted","Data":"bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0"} Oct 30 21:21:35.523847 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:35.523423 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef","Type":"ContainerStarted","Data":"0d344d8dcbec9878e93df88198b2c30c616b04533d797010104b0493681a687b"} Oct 30 21:21:35.641299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:35.641235 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.641221118 podStartE2EDuration="2.641221118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:35.640536436 +0000 UTC m=+2723.748032354" watchObservedRunningTime="2026-10-30 21:21:35.641221118 +0000 UTC m=+2723.748717030" Oct 30 21:21:38.255672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.255636 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.452652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.452603 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.602343 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.602240 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.742955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.742910 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4"] Oct 30 21:21:38.747701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.747663 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.752168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.752133 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.805707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.805670 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"s3-secret\"" Oct 30 21:21:38.852481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.852377 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 21:21:38.852481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.852460 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-dgbvr\"" Oct 30 21:21:38.852669 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.852460 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 30 21:21:38.852766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.852751 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 30 21:21:38.857192 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.856941 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4"] Oct 30 21:21:38.866816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.866771 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.866816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.866820 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.867027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.866841 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrj4\" (UniqueName: \"kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.867027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.866939 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.867027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.867017 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.905134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.905092 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.967577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.967527 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.967795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.967592 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.967795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.967612 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrj4\" (UniqueName: \"kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.967795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.967630 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.967795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.967666 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.968122 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.968091 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.968331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.968306 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.968469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.968450 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:38.970116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:38.970096 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:39.030607 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.030571 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrj4\" (UniqueName: \"kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4\") pod \"tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:39.063296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.063261 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:39.084842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.084802 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:39.253924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.253882 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:39.472919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.472873 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:39.491713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.491681 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4"] Oct 30 21:21:39.494958 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:21:39.494925 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf245ed9b_e283_4563_bd3a_904345da3abc.slice/crio-5defdecb6343b94273e30c87a25a5e471c4ed5b5aa86ea06550335815388b529 WatchSource:0}: Error finding container 5defdecb6343b94273e30c87a25a5e471c4ed5b5aa86ea06550335815388b529: Status 404 returned error can't find the container with id 5defdecb6343b94273e30c87a25a5e471c4ed5b5aa86ea06550335815388b529 Oct 30 21:21:39.546250 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.546209 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" event={"ID":"f245ed9b-e283-4563-bd3a-904345da3abc","Type":"ContainerStarted","Data":"5defdecb6343b94273e30c87a25a5e471c4ed5b5aa86ea06550335815388b529"} Oct 30 21:21:39.688971 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.688919 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2"] Oct 30 21:21:39.693876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.693851 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.746737 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.746631 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 21:21:39.746983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.746964 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 21:21:39.747085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.747000 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 21:21:39.747085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.747063 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 21:21:39.747231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.747216 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 21:21:39.747288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.747226 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-vnmrf\"" Oct 30 21:21:39.753586 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.753541 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2"] Oct 30 21:21:39.775262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775214 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v726g\" (UniqueName: \"kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.775447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775275 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.775447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775392 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.775529 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775481 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.775620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775584 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.775664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775652 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.775707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.775694 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876466 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876527 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v726g\" (UniqueName: \"kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876551 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876568 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876589 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876611 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.876718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.876640 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.877349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.877317 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.877519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.877378 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.877519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.877398 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.879279 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.879252 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.879544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.879528 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.879672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.879656 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:39.938728 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:39.938691 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v726g\" (UniqueName: \"kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g\") pod \"tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:40.008695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.008600 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:40.440017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.439941 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2"] Oct 30 21:21:40.444320 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:21:40.444289 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c99fa0_578f_4832_b7ef_8c2990bce659.slice/crio-09d8b25335a03bd7f38c428d96ed17618f706b7e1aac387aa3895350f16a538d WatchSource:0}: Error finding container 09d8b25335a03bd7f38c428d96ed17618f706b7e1aac387aa3895350f16a538d: Status 404 returned error can't find the container with id 09d8b25335a03bd7f38c428d96ed17618f706b7e1aac387aa3895350f16a538d Oct 30 21:21:40.552583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.552546 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" event={"ID":"f245ed9b-e283-4563-bd3a-904345da3abc","Type":"ContainerStarted","Data":"e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135"} Oct 30 21:21:40.553083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.552610 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:21:40.553828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.553792 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" event={"ID":"26c99fa0-578f-4832-b7ef-8c2990bce659","Type":"ContainerStarted","Data":"09d8b25335a03bd7f38c428d96ed17618f706b7e1aac387aa3895350f16a538d"} Oct 30 21:21:40.670373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.670313 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" podStartSLOduration=2.670293889 podStartE2EDuration="2.670293889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:40.669657069 +0000 UTC m=+2728.777153007" watchObservedRunningTime="2026-10-30 21:21:40.670293889 +0000 UTC m=+2728.777789802" Oct 30 21:21:40.762276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.762191 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:40.966922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:40.966890 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:41.414270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.414232 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:21:41.417692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.417668 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.470102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.470065 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 30 21:21:41.477696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.477662 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:21:41.561133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.561029 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" event={"ID":"26c99fa0-578f-4832-b7ef-8c2990bce659","Type":"ContainerStarted","Data":"81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2"} Oct 30 21:21:41.561133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.561100 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" event={"ID":"26c99fa0-578f-4832-b7ef-8c2990bce659","Type":"ContainerStarted","Data":"1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac"} Oct 30 21:21:41.561658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.561246 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:41.561658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.561344 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:41.562988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.562962 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.85:8083/ready\": dial tcp 10.132.0.85:8083: connect: connection refused" Oct 30 21:21:41.563101 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.562957 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.85:8081/ready\": dial tcp 10.132.0.85:8081: connect: connection refused" Oct 30 21:21:41.592368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.592327 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.592520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.592383 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.592520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.592511 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.592589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.592552 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2c5m\" (UniqueName: \"kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.686076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.685781 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:41.687258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.687207 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" podStartSLOduration=2.687192397 podStartE2EDuration="2.687192397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:41.6845464 +0000 UTC m=+2729.792042312" watchObservedRunningTime="2026-10-30 21:21:41.687192397 +0000 UTC m=+2729.794688309" Oct 30 21:21:41.693422 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.693365 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.693631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.693482 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2c5m\" (UniqueName: \"kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.693713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.693666 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.693779 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.693710 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.694703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.694676 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.694842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.694824 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.696577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.696557 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:41.755448 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:41.755393 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2c5m\" (UniqueName: \"kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:42.566319 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:42.566276 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.85:8083/ready\": dial tcp 10.132.0.85:8083: connect: connection refused" Oct 30 21:21:42.827276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:42.827181 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:43.572082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:43.572045 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:44.315060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:44.315023 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2"] Oct 30 21:21:45.578227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:45.578180 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway" containerID="cri-o://1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac" gracePeriod=30 Oct 30 21:21:45.578227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:45.578209 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway-opa" containerID="cri-o://81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2" gracePeriod=30 Oct 30 21:21:45.832917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:45.832827 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:45.937466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:45.937400 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.034876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.034839 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.035061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.034980 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.087204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.087113 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") DevicePath \"csi-113b31927c29ab22b082280e8c62c14f67a0caad66f0a4302e0e79216886b34c\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.344344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.344258 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/563b42616d4c553b953dc281533e8043ca5a0d0fb5ee950eeb28cb0cfdb34f6b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.407078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.407040 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.532161 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.532117 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:46.587016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.586882 4988 generic.go:356] "Generic (PLEG): container finished" podID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerID="81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2" exitCode=0 Oct 30 21:21:46.587016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.586963 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" event={"ID":"26c99fa0-578f-4832-b7ef-8c2990bce659","Type":"ContainerDied","Data":"81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2"} Oct 30 21:21:46.826953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.826926 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:46.942890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.942798 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.942890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.942849 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.943140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.942898 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g\" (UniqueName: \"kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.943140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.942938 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.943140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.942962 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.943140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.942999 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.943140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.943016 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle\") pod \"26c99fa0-578f-4832-b7ef-8c2990bce659\" (UID: \"26c99fa0-578f-4832-b7ef-8c2990bce659\") " Oct 30 21:21:46.943498 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.943333 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:46.943498 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.943486 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:46.943634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.943491 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:21:46.945186 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.945159 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:46.945525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.945502 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:46.945614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.945583 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "kube-api-access-v726g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:21:46.949785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:46.949756 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant" pod "26c99fa0-578f-4832-b7ef-8c2990bce659" (UID: "26c99fa0-578f-4832-b7ef-8c2990bce659"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:21:47.000489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.000457 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:21:47.006499 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:21:47.006462 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71597d5_c5b3_404a_a21f_1a134066ba9c.slice/crio-6b37cc7dcc2467caef569ec760562a28b6199e215307f7198cb71ac2d508ef9e WatchSource:0}: Error finding container 6b37cc7dcc2467caef569ec760562a28b6199e215307f7198cb71ac2d508ef9e: Status 404 returned error can't find the container with id 6b37cc7dcc2467caef569ec760562a28b6199e215307f7198cb71ac2d508ef9e Oct 30 21:21:47.044648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044617 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.044648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044647 4988 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.044840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044657 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v726g\" (UniqueName: \"kubernetes.io/projected/26c99fa0-578f-4832-b7ef-8c2990bce659-kube-api-access-v726g\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.044840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044666 4988 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-tenant\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.044840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044676 4988 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/26c99fa0-578f-4832-b7ef-8c2990bce659-serving-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.044840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044686 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.044840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.044695 4988 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/26c99fa0-578f-4832-b7ef-8c2990bce659-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:21:47.593404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.593369 4988 generic.go:356] "Generic (PLEG): container finished" podID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerID="1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac" exitCode=0 Oct 30 21:21:47.593898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.593436 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" Oct 30 21:21:47.593898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.593447 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" event={"ID":"26c99fa0-578f-4832-b7ef-8c2990bce659","Type":"ContainerDied","Data":"1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac"} Oct 30 21:21:47.593898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.593513 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2" event={"ID":"26c99fa0-578f-4832-b7ef-8c2990bce659","Type":"ContainerDied","Data":"09d8b25335a03bd7f38c428d96ed17618f706b7e1aac387aa3895350f16a538d"} Oct 30 21:21:47.593898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.593545 4988 scope.go:122] "RemoveContainer" containerID="81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2" Oct 30 21:21:47.595374 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.595349 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d71597d5-c5b3-404a-a21f-1a134066ba9c","Type":"ContainerStarted","Data":"b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e"} Oct 30 21:21:47.595507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.595399 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d71597d5-c5b3-404a-a21f-1a134066ba9c","Type":"ContainerStarted","Data":"6b37cc7dcc2467caef569ec760562a28b6199e215307f7198cb71ac2d508ef9e"} Oct 30 21:21:47.595588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.595570 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:21:47.603041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.603022 4988 scope.go:122] "RemoveContainer" containerID="1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac" Oct 30 21:21:47.612107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.612085 4988 scope.go:122] "RemoveContainer" containerID="81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2" Oct 30 21:21:47.612469 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:21:47.612442 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2\": container with ID starting with 81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2 not found: ID does not exist" containerID="81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2" Oct 30 21:21:47.612558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.612480 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2"} err="failed to get container status \"81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2\": rpc error: code = NotFound desc = could not find container \"81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2\": container with ID starting with 81a31b341f2d3a82920f865471a051f273786f8fa3c9d19d315dd4bcb8371bc2 not found: ID does not exist" Oct 30 21:21:47.612558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.612505 4988 scope.go:122] "RemoveContainer" containerID="1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac" Oct 30 21:21:47.612805 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:21:47.612786 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac\": container with ID starting with 1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac not found: ID does not exist" containerID="1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac" Oct 30 21:21:47.612845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.612812 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac"} err="failed to get container status \"1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac\": rpc error: code = NotFound desc = could not find container \"1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac\": container with ID starting with 1db76b1f4da0342f5c44b053d2b8f506fc55c7cf4de1260d9e604f3c91e0adac not found: ID does not exist" Oct 30 21:21:47.765897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.765835 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=9.765818596999999 podStartE2EDuration="9.765818597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:47.76482856 +0000 UTC m=+2735.872324472" watchObservedRunningTime="2026-10-30 21:21:47.765818597 +0000 UTC m=+2735.873314510" Oct 30 21:21:47.877338 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.877255 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2"] Oct 30 21:21:47.881859 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:47.881830 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-b57f5f47c-b9fr2"] Oct 30 21:21:48.015753 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:21:48.015722 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" path="/var/lib/kubelet/pods/26c99fa0-578f-4832-b7ef-8c2990bce659/volumes" Oct 30 21:21:56.395846 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:21:56.395764 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:22:01.625224 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:01.625186 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:22:04.978182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:04.978140 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:08.607287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:08.607237 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:22:17.192093 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:17.192051 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:18.604976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:18.604934 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:22:26.395667 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:22:26.395628 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:22:28.605900 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:28.605868 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:22:34.599360 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:34.599305 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:34.743888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:34.743848 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:44.138725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:44.138683 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:44.333875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:44.333834 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:46.008214 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.008178 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:46.008741 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.008378 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" containerName="otc-container" containerID="cri-o://bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0" gracePeriod=30 Oct 30 21:22:46.259795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.259724 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:46.339745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.339705 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:46.349689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.349658 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs\") pod \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " Oct 30 21:22:46.349805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.349718 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal\") pod \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " Oct 30 21:22:46.349805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.349768 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj\" (UniqueName: \"kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj\") pod \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " Oct 30 21:22:46.349900 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.349809 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs\") pod \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\" (UID: \"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef\") " Oct 30 21:22:46.350118 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.350091 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal" pod "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" (UID: "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:46.351970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.351945 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs" pod "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" (UID: "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:46.352060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.351945 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj" pod "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" (UID: "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef"). InnerVolumeSpecName "kube-api-access-s59fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:46.352060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.351987 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs" pod "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" (UID: "c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:46.450820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.450772 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:46.450820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.450810 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:46.450820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.450821 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59fj\" (UniqueName: \"kubernetes.io/projected/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-kube-api-access-s59fj\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:46.450820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.450831 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:46.484787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.484750 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:46.853793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.853759 4988 generic.go:356] "Generic (PLEG): container finished" podID="c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" containerID="bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0" exitCode=0 Oct 30 21:22:46.853987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.853809 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:46.853987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.853821 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef","Type":"ContainerDied","Data":"bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0"} Oct 30 21:22:46.853987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.853864 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef","Type":"ContainerDied","Data":"0d344d8dcbec9878e93df88198b2c30c616b04533d797010104b0493681a687b"} Oct 30 21:22:46.853987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.853886 4988 scope.go:122] "RemoveContainer" containerID="bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0" Oct 30 21:22:46.862667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.862643 4988 scope.go:122] "RemoveContainer" containerID="bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0" Oct 30 21:22:46.862925 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:22:46.862905 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0\": container with ID starting with bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0 not found: ID does not exist" containerID="bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0" Oct 30 21:22:46.862977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:46.862935 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0"} err="failed to get container status \"bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0\": rpc error: code = NotFound desc = could not find container \"bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0\": container with ID starting with bf5a96cdc6eb5bbd76c4ae87919dca4f5adc0a0de786297f1f540ef4a25cebf0 not found: ID does not exist" Oct 30 21:22:47.017228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.017196 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:47.067035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.067001 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:47.097695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.097666 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:47.098004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.097992 4988 cpu_manager.go:414] "containerMap: removing container" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway-opa" Oct 30 21:22:47.098047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098005 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway-opa" Oct 30 21:22:47.098047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098023 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" containerName="otc-container" Oct 30 21:22:47.098047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098029 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" containerName="otc-container" Oct 30 21:22:47.098146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098049 4988 cpu_manager.go:414] "containerMap: removing container" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway" Oct 30 21:22:47.098146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098054 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway" Oct 30 21:22:47.098146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098118 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway-opa" Oct 30 21:22:47.098146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098126 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" containerName="otc-container" Oct 30 21:22:47.098146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.098133 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c99fa0-578f-4832-b7ef-8c2990bce659" containerName="tempo-gateway" Oct 30 21:22:47.102395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.102376 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.155800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.155769 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 21:22:47.155800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.155771 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:22:47.155800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.155771 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-phj8t\"" Oct 30 21:22:47.156096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.155835 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:22:47.157076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.157045 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.157076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.157078 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.157248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.157148 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9kg\" (UniqueName: \"kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.157248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.157188 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.163547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.162077 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:47.258276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.258232 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9kg\" (UniqueName: \"kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.258526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.258322 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.258526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.258369 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.258526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.258398 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.259161 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.259134 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.260879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.260853 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.260960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.260915 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.320086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.320046 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9kg\" (UniqueName: \"kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg\") pod \"data-science-collector-collector-1\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.415238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.415202 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:47.790091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.790008 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:47.793375 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:22:47.793346 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36494670_00dd_4b21_a59c_86ebb2d34cc4.slice/crio-dcf99cfdc74b87183ce587e92e251d793e3313e4e5b01d11c0f1246e2042a66c WatchSource:0}: Error finding container dcf99cfdc74b87183ce587e92e251d793e3313e4e5b01d11c0f1246e2042a66c: Status 404 returned error can't find the container with id dcf99cfdc74b87183ce587e92e251d793e3313e4e5b01d11c0f1246e2042a66c Oct 30 21:22:47.795619 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.795576 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:22:47.795701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.795647 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:22:47.861729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.861695 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"36494670-00dd-4b21-a59c-86ebb2d34cc4","Type":"ContainerStarted","Data":"c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698"} Oct 30 21:22:47.861956 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.861760 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"36494670-00dd-4b21-a59c-86ebb2d34cc4","Type":"ContainerStarted","Data":"dcf99cfdc74b87183ce587e92e251d793e3313e4e5b01d11c0f1246e2042a66c"} Oct 30 21:22:47.980374 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:47.980322 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=0.980306731 podStartE2EDuration="980.306731ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:47.979395125 +0000 UTC m=+2796.086891040" watchObservedRunningTime="2026-10-30 21:22:47.980306731 +0000 UTC m=+2796.087802642" Oct 30 21:22:48.017175 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:48.017134 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef" path="/var/lib/kubelet/pods/c502aa51-5e81-4c4f-b9a3-3fd5a753e2ef/volumes" Oct 30 21:22:55.429041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.429004 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:55.429679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.429219 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="36494670-00dd-4b21-a59c-86ebb2d34cc4" containerName="otc-container" containerID="cri-o://c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698" gracePeriod=30 Oct 30 21:22:55.445336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.445302 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:55.677518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.677490 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:55.733111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.733022 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal\") pod \"36494670-00dd-4b21-a59c-86ebb2d34cc4\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " Oct 30 21:22:55.733111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.733069 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs\") pod \"36494670-00dd-4b21-a59c-86ebb2d34cc4\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " Oct 30 21:22:55.733111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.733097 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg\" (UniqueName: \"kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg\") pod \"36494670-00dd-4b21-a59c-86ebb2d34cc4\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " Oct 30 21:22:55.733389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.733167 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs\") pod \"36494670-00dd-4b21-a59c-86ebb2d34cc4\" (UID: \"36494670-00dd-4b21-a59c-86ebb2d34cc4\") " Oct 30 21:22:55.733516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.733486 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal" pod "36494670-00dd-4b21-a59c-86ebb2d34cc4" (UID: "36494670-00dd-4b21-a59c-86ebb2d34cc4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:55.735298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.735256 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs" pod "36494670-00dd-4b21-a59c-86ebb2d34cc4" (UID: "36494670-00dd-4b21-a59c-86ebb2d34cc4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:55.735298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.735267 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg" pod "36494670-00dd-4b21-a59c-86ebb2d34cc4" (UID: "36494670-00dd-4b21-a59c-86ebb2d34cc4"). InnerVolumeSpecName "kube-api-access-bf9kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:55.735480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.735335 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs" pod "36494670-00dd-4b21-a59c-86ebb2d34cc4" (UID: "36494670-00dd-4b21-a59c-86ebb2d34cc4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:55.834547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.834512 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:55.834547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.834544 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/36494670-00dd-4b21-a59c-86ebb2d34cc4-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:55.834547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.834554 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/36494670-00dd-4b21-a59c-86ebb2d34cc4-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:55.834784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.834563 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9kg\" (UniqueName: \"kubernetes.io/projected/36494670-00dd-4b21-a59c-86ebb2d34cc4-kube-api-access-bf9kg\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:55.846064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.846027 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:55.897558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.897524 4988 generic.go:356] "Generic (PLEG): container finished" podID="36494670-00dd-4b21-a59c-86ebb2d34cc4" containerID="c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698" exitCode=0 Oct 30 21:22:55.897740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.897568 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:22:55.897740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.897578 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"36494670-00dd-4b21-a59c-86ebb2d34cc4","Type":"ContainerDied","Data":"c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698"} Oct 30 21:22:55.897740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.897619 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"36494670-00dd-4b21-a59c-86ebb2d34cc4","Type":"ContainerDied","Data":"dcf99cfdc74b87183ce587e92e251d793e3313e4e5b01d11c0f1246e2042a66c"} Oct 30 21:22:55.897740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.897641 4988 scope.go:122] "RemoveContainer" containerID="c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698" Oct 30 21:22:55.907335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.907311 4988 scope.go:122] "RemoveContainer" containerID="c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698" Oct 30 21:22:55.907673 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:22:55.907648 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698\": container with ID starting with c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698 not found: ID does not exist" containerID="c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698" Oct 30 21:22:55.907757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:55.907680 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698"} err="failed to get container status \"c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698\": rpc error: code = NotFound desc = could not find container \"c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698\": container with ID starting with c2766bcf923c7467a3556361e6b644928f4f93439c86a161a3d1c36b3d83e698 not found: ID does not exist" Oct 30 21:22:56.073941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.073852 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:56.077817 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.077790 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:22:56.311853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.311817 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:56.399377 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:22:56.399339 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:22:56.406976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.406938 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:22:56.407218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.407168 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" containerID="cri-o://b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e" gracePeriod=30 Oct 30 21:22:56.434745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.434708 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4"] Oct 30 21:22:56.435162 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.434923 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" containerID="cri-o://e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135" gracePeriod=30 Oct 30 21:22:56.490452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.487106 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:56.630125 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:56.630084 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:58.015543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:58.015503 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36494670-00dd-4b21-a59c-86ebb2d34cc4" path="/var/lib/kubelet/pods/36494670-00dd-4b21-a59c-86ebb2d34cc4/volumes" Oct 30 21:22:58.605912 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:58.605860 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:22:59.464922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.464896 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:22:59.570656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.570568 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"d71597d5-c5b3-404a-a21f-1a134066ba9c\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " Oct 30 21:22:59.570656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.570629 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls\") pod \"d71597d5-c5b3-404a-a21f-1a134066ba9c\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " Oct 30 21:22:59.570914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.570689 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf\") pod \"d71597d5-c5b3-404a-a21f-1a134066ba9c\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " Oct 30 21:22:59.570914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.570718 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m\" (UniqueName: \"kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m\") pod \"d71597d5-c5b3-404a-a21f-1a134066ba9c\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " Oct 30 21:22:59.570914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.570771 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle\") pod \"d71597d5-c5b3-404a-a21f-1a134066ba9c\" (UID: \"d71597d5-c5b3-404a-a21f-1a134066ba9c\") " Oct 30 21:22:59.571218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.571189 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf" pod "d71597d5-c5b3-404a-a21f-1a134066ba9c" (UID: "d71597d5-c5b3-404a-a21f-1a134066ba9c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:59.571310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.571253 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle" pod "d71597d5-c5b3-404a-a21f-1a134066ba9c" (UID: "d71597d5-c5b3-404a-a21f-1a134066ba9c"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:59.573038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.573003 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m" pod "d71597d5-c5b3-404a-a21f-1a134066ba9c" (UID: "d71597d5-c5b3-404a-a21f-1a134066ba9c"). InnerVolumeSpecName "kube-api-access-k2c5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:59.573038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.573023 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls" pod "d71597d5-c5b3-404a-a21f-1a134066ba9c" (UID: "d71597d5-c5b3-404a-a21f-1a134066ba9c"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:59.573539 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.573519 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62" pod "d71597d5-c5b3-404a-a21f-1a134066ba9c" (UID: "d71597d5-c5b3-404a-a21f-1a134066ba9c"). InnerVolumeSpecName "pvc-8413c93a-423e-4f5d-b17d-1465d72f5969". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:22:59.672000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.671956 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:59.672000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.671990 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2c5m\" (UniqueName: \"kubernetes.io/projected/d71597d5-c5b3-404a-a21f-1a134066ba9c-kube-api-access-k2c5m\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:59.672000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.672001 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:59.672258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.672033 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:22:59.672258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.672044 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d71597d5-c5b3-404a-a21f-1a134066ba9c-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:22:59.682576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.682553 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-8413c93a-423e-4f5d-b17d-1465d72f5969" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:22:59.773068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.773027 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-113b31927c29ab22b082280e8c62c14f67a0caad66f0a4302e0e79216886b34c\"" Oct 30 21:22:59.919479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.919437 4988 generic.go:356] "Generic (PLEG): container finished" podID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerID="b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e" exitCode=0 Oct 30 21:22:59.919695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.919552 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:22:59.919695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.919583 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d71597d5-c5b3-404a-a21f-1a134066ba9c","Type":"ContainerDied","Data":"b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e"} Oct 30 21:22:59.919695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.919655 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d71597d5-c5b3-404a-a21f-1a134066ba9c","Type":"ContainerDied","Data":"6b37cc7dcc2467caef569ec760562a28b6199e215307f7198cb71ac2d508ef9e"} Oct 30 21:22:59.919695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.919684 4988 scope.go:122] "RemoveContainer" containerID="b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e" Oct 30 21:22:59.929786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.929766 4988 scope.go:122] "RemoveContainer" containerID="b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e" Oct 30 21:22:59.930120 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:22:59.930101 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e\": container with ID starting with b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e not found: ID does not exist" containerID="b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e" Oct 30 21:22:59.930172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:22:59.930129 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e"} err="failed to get container status \"b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e\": rpc error: code = NotFound desc = could not find container \"b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e\": container with ID starting with b1f291dc0888bbf308b277b40e9b206c23427814552cd196207626804a57f05e not found: ID does not exist" Oct 30 21:23:00.083913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:00.083876 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:00.088019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:00.087986 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:01.829586 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:01.829533 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:23:02.015477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:02.015440 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" path="/var/lib/kubelet/pods/d71597d5-c5b3-404a-a21f-1a134066ba9c/volumes" Oct 30 21:23:11.566310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:11.566257 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:23:17.489197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:17.489161 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:21.566157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:21.566119 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:23:21.566580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:21.566223 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:23:24.810942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:24.810853 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:26.395027 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:23:26.394990 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:23:26.582192 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.582167 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:23:26.706466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706375 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle\") pod \"f245ed9b-e283-4563-bd3a-904345da3abc\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " Oct 30 21:23:26.706636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706460 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls\") pod \"f245ed9b-e283-4563-bd3a-904345da3abc\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " Oct 30 21:23:26.706636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706491 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4\" (UniqueName: \"kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4\") pod \"f245ed9b-e283-4563-bd3a-904345da3abc\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " Oct 30 21:23:26.706636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706519 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage\") pod \"f245ed9b-e283-4563-bd3a-904345da3abc\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " Oct 30 21:23:26.706636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706606 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf\") pod \"f245ed9b-e283-4563-bd3a-904345da3abc\" (UID: \"f245ed9b-e283-4563-bd3a-904345da3abc\") " Oct 30 21:23:26.706942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706879 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle" pod "f245ed9b-e283-4563-bd3a-904345da3abc" (UID: "f245ed9b-e283-4563-bd3a-904345da3abc"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:26.706942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.706882 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage" pod "f245ed9b-e283-4563-bd3a-904345da3abc" (UID: "f245ed9b-e283-4563-bd3a-904345da3abc"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:26.707194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.707169 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf" pod "f245ed9b-e283-4563-bd3a-904345da3abc" (UID: "f245ed9b-e283-4563-bd3a-904345da3abc"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:26.708831 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.708802 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls" pod "f245ed9b-e283-4563-bd3a-904345da3abc" (UID: "f245ed9b-e283-4563-bd3a-904345da3abc"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:26.708960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.708888 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4" pod "f245ed9b-e283-4563-bd3a-904345da3abc" (UID: "f245ed9b-e283-4563-bd3a-904345da3abc"). InnerVolumeSpecName "kube-api-access-tnrj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:26.807951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.807911 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:26.807951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.807941 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:26.807951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.807952 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:26.807951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.807961 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrj4\" (UniqueName: \"kubernetes.io/projected/f245ed9b-e283-4563-bd3a-904345da3abc-kube-api-access-tnrj4\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:26.808227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:26.807970 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f245ed9b-e283-4563-bd3a-904345da3abc-tempo-tmp-storage\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:27.048430 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.048326 4988 generic.go:356] "Generic (PLEG): container finished" podID="f245ed9b-e283-4563-bd3a-904345da3abc" containerID="e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135" exitCode=137 Oct 30 21:23:27.048430 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.048372 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" Oct 30 21:23:27.048826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.048376 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" event={"ID":"f245ed9b-e283-4563-bd3a-904345da3abc","Type":"ContainerDied","Data":"e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135"} Oct 30 21:23:27.048826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.048509 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4" event={"ID":"f245ed9b-e283-4563-bd3a-904345da3abc","Type":"ContainerDied","Data":"5defdecb6343b94273e30c87a25a5e471c4ed5b5aa86ea06550335815388b529"} Oct 30 21:23:27.048826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.048542 4988 scope.go:122] "RemoveContainer" containerID="e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135" Oct 30 21:23:27.057387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.057366 4988 scope.go:122] "RemoveContainer" containerID="e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135" Oct 30 21:23:27.057718 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:23:27.057698 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135\": container with ID starting with e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135 not found: ID does not exist" containerID="e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135" Oct 30 21:23:27.057773 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.057729 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135"} err="failed to get container status \"e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135\": rpc error: code = NotFound desc = could not find container \"e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135\": container with ID starting with e6fde10809d6b7a587c4ef63e988e726a75c45162f472c3cbf1ca99ad945c135 not found: ID does not exist" Oct 30 21:23:27.212209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.212172 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4"] Oct 30 21:23:27.215528 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:27.215504 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6c998ffcd4-pfbc4"] Oct 30 21:23:28.015523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:28.015480 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" path="/var/lib/kubelet/pods/f245ed9b-e283-4563-bd3a-904345da3abc/volumes" Oct 30 21:23:29.236314 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:29.236273 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:45.773944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:45.773904 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:45.917049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:45.917009 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:46.147193 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:46.147154 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:46.293932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:46.293889 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:47.901940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:47.901901 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:48.061041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.060995 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:48.130335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130303 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:23:48.130649 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130636 4988 cpu_manager.go:414] "containerMap: removing container" podUID="36494670-00dd-4b21-a59c-86ebb2d34cc4" containerName="otc-container" Oct 30 21:23:48.130697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130651 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36494670-00dd-4b21-a59c-86ebb2d34cc4" containerName="otc-container" Oct 30 21:23:48.130697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130664 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" Oct 30 21:23:48.130697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130669 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" Oct 30 21:23:48.130697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130684 4988 cpu_manager.go:414] "containerMap: removing container" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" Oct 30 21:23:48.130697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130690 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" Oct 30 21:23:48.130856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130750 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f245ed9b-e283-4563-bd3a-904345da3abc" containerName="tempo" Oct 30 21:23:48.130856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130763 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="36494670-00dd-4b21-a59c-86ebb2d34cc4" containerName="otc-container" Oct 30 21:23:48.130856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.130769 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71597d5-c5b3-404a-a21f-1a134066ba9c" containerName="tempo" Oct 30 21:23:48.133759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.133731 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.160473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.160351 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.160473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.160405 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.160473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.160447 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dnk\" (UniqueName: \"kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.160720 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.160557 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.213484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.213430 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:48.236676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.236643 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:23:48.237115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.237091 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:23:48.237242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.237176 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:23:48.261510 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.261479 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.261671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.261526 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.261671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.261558 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.261671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.261577 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dnk\" (UniqueName: \"kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.263993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.263966 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.263993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.263991 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.278309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.278279 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 21:23:48.282945 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.282915 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.285949 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.285925 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:23:48.286070 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.285966 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-nfqbf\"" Oct 30 21:23:48.287553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.287532 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:23:48.323104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.323071 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dnk\" (UniqueName: \"kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk\") pod \"data-science-collector-collector-1\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.448131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.448053 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:23:48.824548 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.824472 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:23:48.827782 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:23:48.827753 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc968c75b_4054_404c_9215_70a2f1ab8c26.slice/crio-604d4fe4fad61b08ff94b016d3a26983ad1ebc5983b5bb25849264235b3ca8c5 WatchSource:0}: Error finding container 604d4fe4fad61b08ff94b016d3a26983ad1ebc5983b5bb25849264235b3ca8c5: Status 404 returned error can't find the container with id 604d4fe4fad61b08ff94b016d3a26983ad1ebc5983b5bb25849264235b3ca8c5 Oct 30 21:23:48.830051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.830020 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:48.830158 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:48.830083 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:49.142292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:49.142252 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c968c75b-4054-404c-9215-70a2f1ab8c26","Type":"ContainerStarted","Data":"304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5"} Oct 30 21:23:49.142844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:49.142317 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c968c75b-4054-404c-9215-70a2f1ab8c26","Type":"ContainerStarted","Data":"604d4fe4fad61b08ff94b016d3a26983ad1ebc5983b5bb25849264235b3ca8c5"} Oct 30 21:23:49.261268 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:49.261210 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.2611953630000001 podStartE2EDuration="1.261195363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:49.260103803 +0000 UTC m=+2857.367599728" watchObservedRunningTime="2026-10-30 21:23:49.261195363 +0000 UTC m=+2857.368691334" Oct 30 21:23:51.098319 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.098249 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.237640 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.237598 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:51.241071 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.241044 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.277919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.277883 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.293622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.293582 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 21:23:51.343162 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.343124 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-jflh2\"" Oct 30 21:23:51.343162 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.343122 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 30 21:23:51.343394 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.343188 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\"" Oct 30 21:23:51.343394 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.343193 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 30 21:23:51.344558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.344536 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:51.426605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.426561 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.482762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.482725 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.482995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.482783 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.482995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.482891 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.482995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.482941 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.483121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.483011 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjp8\" (UniqueName: \"kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.572479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.572392 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.584292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.584248 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.584531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.584303 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.584531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.584332 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjp8\" (UniqueName: \"kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.584531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.584363 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.584531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.584398 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.584781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.584616 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj"] Oct 30 21:23:51.585132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.585101 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.585439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.585382 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.587090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.587060 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.587198 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.587092 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.589446 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.589426 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.641713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.641676 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 30 21:23:51.646099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.646069 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjp8\" (UniqueName: \"kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:51.647578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.647552 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj"] Oct 30 21:23:51.685642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.685544 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.685642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.685609 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.685850 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.685684 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.685850 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.685822 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmm7\" (UniqueName: \"kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.685977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.685879 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.685977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.685934 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.740613 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.740571 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:51.786370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786324 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmm7\" (UniqueName: \"kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.786601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786390 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.786601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786464 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.786601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786510 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.786601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786564 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.786601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786586 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.786985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.786961 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.787251 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.787229 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.787392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.787374 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.789163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.789134 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.789327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.789305 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.847821 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.847790 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmm7\" (UniqueName: \"kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7\") pod \"tempo-data-science-tempostack-query-frontend-7bb787577-d76lj\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.911891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.911854 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:51.915068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:51.915031 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:52.085537 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.085496 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:52.340521 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.340394 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj"] Oct 30 21:23:52.345135 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:23:52.345107 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0ac17f_620d_4da9_803c_3208b2cec95c.slice/crio-a7bfa3abed6aa14838b5e7d4cea3b3ec6e6ab004f6745bd057dca03529c0d709 WatchSource:0}: Error finding container a7bfa3abed6aa14838b5e7d4cea3b3ec6e6ab004f6745bd057dca03529c0d709: Status 404 returned error can't find the container with id a7bfa3abed6aa14838b5e7d4cea3b3ec6e6ab004f6745bd057dca03529c0d709 Oct 30 21:23:52.455574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.455534 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:52.539085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.539047 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g"] Oct 30 21:23:52.543642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.543623 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.596612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.596531 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 21:23:52.596796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.596593 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 21:23:52.596867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.596653 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 21:23:52.598953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.598934 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-jcvnx\"" Oct 30 21:23:52.599046 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.598984 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 21:23:52.647531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.647486 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g"] Oct 30 21:23:52.695309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695278 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.695509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695319 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.695509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695351 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhw7\" (UniqueName: \"kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.695509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695451 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.695509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695497 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.695678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695527 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.695678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.695554 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.696504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.696480 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 21:23:52.796289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796254 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.796535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796305 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhw7\" (UniqueName: \"kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.796535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796440 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.796535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796492 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.796535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796522 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.796756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796614 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.796756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.796696 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.797138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.797105 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.797276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.797254 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.797357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.797254 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.799053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.799024 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.799053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.799034 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.799191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.799165 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.858117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.858043 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhw7\" (UniqueName: \"kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7\") pod \"tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:52.861717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:52.861685 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:53.161166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.161127 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" event={"ID":"5f0ac17f-620d-4da9-803c-3208b2cec95c","Type":"ContainerStarted","Data":"3be3fa218b82a436590bf579774c986db0d42a665581f8f589b4b7cca1b63202"} Oct 30 21:23:53.161390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.161198 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:23:53.161390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.161217 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" event={"ID":"5f0ac17f-620d-4da9-803c-3208b2cec95c","Type":"ContainerStarted","Data":"a7bfa3abed6aa14838b5e7d4cea3b3ec6e6ab004f6745bd057dca03529c0d709"} Oct 30 21:23:53.280615 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.280554 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" podStartSLOduration=2.280537628 podStartE2EDuration="2.280537628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:53.279406809 +0000 UTC m=+2861.386936324" watchObservedRunningTime="2026-10-30 21:23:53.280537628 +0000 UTC m=+2861.388033539" Oct 30 21:23:53.288917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.288880 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g"] Oct 30 21:23:53.292203 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:23:53.292172 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429687ca_16c3_4e87_8914_84f84ee3f22d.slice/crio-178cf05e86010c4ac360241fd95e2ed7e70daa9b52b7e548d96df8484d54c64b WatchSource:0}: Error finding container 178cf05e86010c4ac360241fd95e2ed7e70daa9b52b7e548d96df8484d54c64b: Status 404 returned error can't find the container with id 178cf05e86010c4ac360241fd95e2ed7e70daa9b52b7e548d96df8484d54c64b Oct 30 21:23:53.327262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.327223 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:53.339989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.339955 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2"] Oct 30 21:23:53.344597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.344577 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.380112 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:23:53.380072 4988 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="071cc96d-0350-4ade-b1cd-6ff87dca9fcf" Oct 30 21:23:53.399271 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.399235 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 30 21:23:53.404026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.403993 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2"] Oct 30 21:23:53.504436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.504383 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.504604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.504487 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdns\" (UniqueName: \"kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.504604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.504531 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.504604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.504567 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.504734 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.504671 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.605505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.605467 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.605729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.605520 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdns\" (UniqueName: \"kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.605729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.605547 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.605729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.605574 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.605729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.605706 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.606044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.606022 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.606360 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.606341 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.606360 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.606354 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.624647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.624609 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.628144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.628114 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw"] Oct 30 21:23:53.632886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.632864 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.669073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.669040 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdns\" (UniqueName: \"kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns\") pod \"tempo-data-science-tempostack-distributor-b8c4ff99-99bx2\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.690276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.690182 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26"] Oct 30 21:23:53.693861 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.693834 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.694958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.694928 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw"] Oct 30 21:23:53.733007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.732971 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 30 21:23:53.755806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.755772 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26"] Oct 30 21:23:53.808456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808392 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.808456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808458 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.808681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808518 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.808681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808631 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.808681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808663 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.808782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808683 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.808782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808736 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.808782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808773 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmq54\" (UniqueName: \"kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.808874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808823 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.808874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808866 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7dg\" (UniqueName: \"kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.808939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808887 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.808973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.808944 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.832899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.832862 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 30 21:23:53.910137 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910098 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.910137 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910141 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.910371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910313 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.910371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910345 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmq54\" (UniqueName: \"kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.910490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910394 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.910490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910434 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7dg\" (UniqueName: \"kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.910646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910591 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.910719 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910676 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.910719 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910705 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.910827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910729 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.910827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910756 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.910827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.910790 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.911140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.911050 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.911651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.911620 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.911849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.911827 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.912021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.912000 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.912367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.912344 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.913075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.912642 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.913459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.913381 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.913808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.913782 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.920806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.920763 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:53.922930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.922900 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.957159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.957064 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:53.974244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.974214 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmq54\" (UniqueName: \"kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54\") pod \"tempo-data-science-tempostack-compactor-557bcbd889-rsj26\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:53.979522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:53.979500 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7dg\" (UniqueName: \"kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg\") pod \"tempo-data-science-tempostack-querier-59776c68c7-bwqtw\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:54.010447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.009499 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:54.167738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.167700 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" event={"ID":"429687ca-16c3-4e87-8914-84f84ee3f22d","Type":"ContainerStarted","Data":"b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66"} Oct 30 21:23:54.167988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.167774 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" event={"ID":"429687ca-16c3-4e87-8914-84f84ee3f22d","Type":"ContainerStarted","Data":"208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717"} Oct 30 21:23:54.167988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.167800 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" event={"ID":"429687ca-16c3-4e87-8914-84f84ee3f22d","Type":"ContainerStarted","Data":"178cf05e86010c4ac360241fd95e2ed7e70daa9b52b7e548d96df8484d54c64b"} Oct 30 21:23:54.167988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.167915 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:54.167988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.167963 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:54.167988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.167973 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:54.169886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.169855 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.132.0.91:8081/ready\": dial tcp 10.132.0.91:8081: connect: connection refused" Oct 30 21:23:54.170004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.169859 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.91:8083/ready\": dial tcp 10.132.0.91:8083: connect: connection refused" Oct 30 21:23:54.176719 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.176695 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:54.267398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.267297 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:54.289659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.289602 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" podStartSLOduration=2.289587173 podStartE2EDuration="2.289587173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:54.287677545 +0000 UTC m=+2862.395173459" watchObservedRunningTime="2026-10-30 21:23:54.289587173 +0000 UTC m=+2862.397083083" Oct 30 21:23:54.314263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.314219 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key\") pod \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " Oct 30 21:23:54.314489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.314309 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf\") pod \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " Oct 30 21:23:54.314489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.314399 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8\" (UniqueName: \"kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8\") pod \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " Oct 30 21:23:54.314489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.314482 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle\") pod \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " Oct 30 21:23:54.314675 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.314545 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls\") pod \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\" (UID: \"071cc96d-0350-4ade-b1cd-6ff87dca9fcf\") " Oct 30 21:23:54.315070 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.315010 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle" pod "071cc96d-0350-4ade-b1cd-6ff87dca9fcf" (UID: "071cc96d-0350-4ade-b1cd-6ff87dca9fcf"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:54.317337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.315724 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf" pod "071cc96d-0350-4ade-b1cd-6ff87dca9fcf" (UID: "071cc96d-0350-4ade-b1cd-6ff87dca9fcf"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:54.317337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.315997 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:54.317337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.316022 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:54.317337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.317159 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key" pod "071cc96d-0350-4ade-b1cd-6ff87dca9fcf" (UID: "071cc96d-0350-4ade-b1cd-6ff87dca9fcf"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:54.319193 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.319159 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls" pod "071cc96d-0350-4ade-b1cd-6ff87dca9fcf" (UID: "071cc96d-0350-4ade-b1cd-6ff87dca9fcf"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:54.319323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.319267 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8" pod "071cc96d-0350-4ade-b1cd-6ff87dca9fcf" (UID: "071cc96d-0350-4ade-b1cd-6ff87dca9fcf"). InnerVolumeSpecName "kube-api-access-8mjp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:54.334235 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.334196 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2"] Oct 30 21:23:54.338292 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:23:54.338252 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c091f0d_d1ee_49b1_b9bf_f2c95aa90a94.slice/crio-a5f2b925dfb4ffe6ff27d5ca6dbc83aa8ad98123c45e76a78ba05431f32faa38 WatchSource:0}: Error finding container a5f2b925dfb4ffe6ff27d5ca6dbc83aa8ad98123c45e76a78ba05431f32faa38: Status 404 returned error can't find the container with id a5f2b925dfb4ffe6ff27d5ca6dbc83aa8ad98123c45e76a78ba05431f32faa38 Oct 30 21:23:54.416527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.416489 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjp8\" (UniqueName: \"kubernetes.io/projected/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-kube-api-access-8mjp8\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:54.416527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.416525 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:54.416527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.416537 4988 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/071cc96d-0350-4ade-b1cd-6ff87dca9fcf-storage-gcs-key\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:54.439712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.439670 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26"] Oct 30 21:23:54.443829 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:23:54.443793 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecefc32_81b3_4831_a88c_a6bef976d270.slice/crio-6cefc057603587d98ca3c4ed3fcb1564d4526ee2b0b640c1d5ca867103bc0cfb WatchSource:0}: Error finding container 6cefc057603587d98ca3c4ed3fcb1564d4526ee2b0b640c1d5ca867103bc0cfb: Status 404 returned error can't find the container with id 6cefc057603587d98ca3c4ed3fcb1564d4526ee2b0b640c1d5ca867103bc0cfb Oct 30 21:23:54.645152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:54.645114 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw"] Oct 30 21:23:54.648379 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:23:54.648343 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd64e775_f161_4439_bf7b_5348c458676b.slice/crio-a26fa20c54ed6db27135a0f93e9bf9c578e3b360080b22a9f04b507a3225926e WatchSource:0}: Error finding container a26fa20c54ed6db27135a0f93e9bf9c578e3b360080b22a9f04b507a3225926e: Status 404 returned error can't find the container with id a26fa20c54ed6db27135a0f93e9bf9c578e3b360080b22a9f04b507a3225926e Oct 30 21:23:55.174290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.174252 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" event={"ID":"9ecefc32-81b3-4831-a88c-a6bef976d270","Type":"ContainerStarted","Data":"3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50"} Oct 30 21:23:55.174523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.174304 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" event={"ID":"9ecefc32-81b3-4831-a88c-a6bef976d270","Type":"ContainerStarted","Data":"6cefc057603587d98ca3c4ed3fcb1564d4526ee2b0b640c1d5ca867103bc0cfb"} Oct 30 21:23:55.174523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.174361 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:23:55.175967 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.175945 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" event={"ID":"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94","Type":"ContainerStarted","Data":"0c7f36666225fde4b6097659faa3527f489fe25a4d39ef3ac689424db3336aff"} Oct 30 21:23:55.176091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.175984 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:23:55.176091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.175999 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" event={"ID":"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94","Type":"ContainerStarted","Data":"a5f2b925dfb4ffe6ff27d5ca6dbc83aa8ad98123c45e76a78ba05431f32faa38"} Oct 30 21:23:55.177454 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.177424 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" event={"ID":"fd64e775-f161-4439-bf7b-5348c458676b","Type":"ContainerStarted","Data":"27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468"} Oct 30 21:23:55.177578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.177467 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:23:55.177578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.177484 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" event={"ID":"fd64e775-f161-4439-bf7b-5348c458676b","Type":"ContainerStarted","Data":"a26fa20c54ed6db27135a0f93e9bf9c578e3b360080b22a9f04b507a3225926e"} Oct 30 21:23:55.177578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.177503 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.177861 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.177839 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.91:8083/ready\": dial tcp 10.132.0.91:8083: connect: connection refused" Oct 30 21:23:55.293608 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.293542 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podStartSLOduration=2.293524917 podStartE2EDuration="2.293524917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:55.292767047 +0000 UTC m=+2863.400262961" watchObservedRunningTime="2026-10-30 21:23:55.293524917 +0000 UTC m=+2863.401020829" Oct 30 21:23:55.468883 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.468790 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:55.474829 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.474799 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:55.553918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.553882 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:55.558514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.558493 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.594204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.594150 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" podStartSLOduration=2.594137909 podStartE2EDuration="2.594137909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:55.592988927 +0000 UTC m=+2863.700484840" watchObservedRunningTime="2026-10-30 21:23:55.594137909 +0000 UTC m=+2863.701633818" Oct 30 21:23:55.611074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.611042 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 30 21:23:55.617238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.617198 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:23:55.711706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.711651 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" podStartSLOduration=2.711636221 podStartE2EDuration="2.711636221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:55.710456987 +0000 UTC m=+2863.817952900" watchObservedRunningTime="2026-10-30 21:23:55.711636221 +0000 UTC m=+2863.819132143" Oct 30 21:23:55.827384 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.827311 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:55.828403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.828377 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.828530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.828499 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.828573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.828532 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.828613 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.828596 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2tc\" (UniqueName: \"kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.828659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.828644 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.929818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.929763 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2tc\" (UniqueName: \"kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.929818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.929831 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.930058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.929860 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.930058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.929898 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.930058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.929919 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.930576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.930549 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.930820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.930800 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.932386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.932362 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.932496 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.932451 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:55.991245 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:55.991206 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2tc\" (UniqueName: \"kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:56.016614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:56.016580 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071cc96d-0350-4ade-b1cd-6ff87dca9fcf" path="/var/lib/kubelet/pods/071cc96d-0350-4ade-b1cd-6ff87dca9fcf/volumes" Oct 30 21:23:56.184332 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:56.184302 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:56.394644 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:23:56.394605 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:23:57.163964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:57.163927 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g"] Oct 30 21:23:58.192687 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:58.192625 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway" containerID="cri-o://208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717" gracePeriod=30 Oct 30 21:23:58.193282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:58.192644 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway-opa" containerID="cri-o://b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66" gracePeriod=30 Oct 30 21:23:59.198525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.198491 4988 generic.go:356] "Generic (PLEG): container finished" podID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerID="b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66" exitCode=0 Oct 30 21:23:59.198908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.198563 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" event={"ID":"429687ca-16c3-4e87-8914-84f84ee3f22d","Type":"ContainerDied","Data":"b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66"} Oct 30 21:23:59.440276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.440247 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:23:59.455388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455287 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.455388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455342 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.455388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455380 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.455709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455451 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.455709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455501 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.455821 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455736 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:59.455869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455812 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:59.455940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.455920 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:59.457546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.457527 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:59.461921 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.461895 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:59.556385 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556340 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7\" (UniqueName: \"kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.556385 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556391 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls\") pod \"429687ca-16c3-4e87-8914-84f84ee3f22d\" (UID: \"429687ca-16c3-4e87-8914-84f84ee3f22d\") " Oct 30 21:23:59.556660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556648 4988 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.556703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556662 4988 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.556703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556670 4988 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-serving-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.556703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556679 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.556703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.556688 4988 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tenant\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.558518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.558493 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:59.558568 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.558494 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7" pod "429687ca-16c3-4e87-8914-84f84ee3f22d" (UID: "429687ca-16c3-4e87-8914-84f84ee3f22d"). InnerVolumeSpecName "kube-api-access-ldhw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:59.657163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.657127 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhw7\" (UniqueName: \"kubernetes.io/projected/429687ca-16c3-4e87-8914-84f84ee3f22d-kube-api-access-ldhw7\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.657163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.657160 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/429687ca-16c3-4e87-8914-84f84ee3f22d-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:59.758217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.758113 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:59.862403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.862369 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:59.959395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.959355 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:23:59.959602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:23:59.959484 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:00.011916 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.011828 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") DevicePath \"csi-113b31927c29ab22b082280e8c62c14f67a0caad66f0a4302e0e79216886b34c\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:00.152547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.152511 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/563b42616d4c553b953dc281533e8043ca5a0d0fb5ee950eeb28cb0cfdb34f6b/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:00.205833 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.205793 4988 generic.go:356] "Generic (PLEG): container finished" podID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerID="208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717" exitCode=0 Oct 30 21:24:00.206322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.205863 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" event={"ID":"429687ca-16c3-4e87-8914-84f84ee3f22d","Type":"ContainerDied","Data":"208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717"} Oct 30 21:24:00.206322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.205907 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" Oct 30 21:24:00.206322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.205942 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g" event={"ID":"429687ca-16c3-4e87-8914-84f84ee3f22d","Type":"ContainerDied","Data":"178cf05e86010c4ac360241fd95e2ed7e70daa9b52b7e548d96df8484d54c64b"} Oct 30 21:24:00.206322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.205957 4988 scope.go:122] "RemoveContainer" containerID="b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66" Oct 30 21:24:00.215925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.215902 4988 scope.go:122] "RemoveContainer" containerID="208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717" Oct 30 21:24:00.218092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.218061 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:00.224977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.224953 4988 scope.go:122] "RemoveContainer" containerID="b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66" Oct 30 21:24:00.225310 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:24:00.225291 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66\": container with ID starting with b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66 not found: ID does not exist" containerID="b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66" Oct 30 21:24:00.225379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.225320 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66"} err="failed to get container status \"b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66\": rpc error: code = NotFound desc = could not find container \"b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66\": container with ID starting with b55b74d408979683e33180f90683d65bf9c43b2bdb2a5e59d63b779358f4fb66 not found: ID does not exist" Oct 30 21:24:00.225379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.225338 4988 scope.go:122] "RemoveContainer" containerID="208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717" Oct 30 21:24:00.225631 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:24:00.225599 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717\": container with ID starting with 208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717 not found: ID does not exist" containerID="208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717" Oct 30 21:24:00.225678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.225634 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717"} err="failed to get container status \"208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717\": rpc error: code = NotFound desc = could not find container \"208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717\": container with ID starting with 208ae867ed68b3588c0b6d92d9be93ce1ea50fa256349839d3dbb906ee83f717 not found: ID does not exist" Oct 30 21:24:00.371514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.371474 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g"] Oct 30 21:24:00.371838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.371818 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:00.379603 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.379570 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d87cbf69-4tk6g"] Oct 30 21:24:00.799242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:00.799204 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:24:00.806238 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:24:00.806208 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a6a590_0e64_4fa1_b0f1_a8ef8cee11d9.slice/crio-a79689a965926f624b9d479ca598ba4c927f03167513feb6a7d8a78aeaf67b7d WatchSource:0}: Error finding container a79689a965926f624b9d479ca598ba4c927f03167513feb6a7d8a78aeaf67b7d: Status 404 returned error can't find the container with id a79689a965926f624b9d479ca598ba4c927f03167513feb6a7d8a78aeaf67b7d Oct 30 21:24:01.212304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:01.212267 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9","Type":"ContainerStarted","Data":"1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60"} Oct 30 21:24:01.212816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:01.212322 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9","Type":"ContainerStarted","Data":"a79689a965926f624b9d479ca598ba4c927f03167513feb6a7d8a78aeaf67b7d"} Oct 30 21:24:01.212816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:01.212433 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:01.331342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:01.331281 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=6.331265627 podStartE2EDuration="6.331265627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:01.330551421 +0000 UTC m=+2869.438047331" watchObservedRunningTime="2026-10-30 21:24:01.331265627 +0000 UTC m=+2869.438761537" Oct 30 21:24:02.015197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:02.015156 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" path="/var/lib/kubelet/pods/429687ca-16c3-4e87-8914-84f84ee3f22d/volumes" Oct 30 21:24:03.704130 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:03.704095 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:14.225981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:14.225951 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:24:16.189086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:16.189049 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:24:16.189576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:16.189252 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:16.189576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:16.189389 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:24:17.007974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:17.007931 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj"] Oct 30 21:24:17.008167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:17.008142 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" containerID="cri-o://3be3fa218b82a436590bf579774c986db0d42a665581f8f589b4b7cca1b63202" gracePeriod=30 Oct 30 21:24:17.838061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:17.837962 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:22.224143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:22.224100 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:24.227170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:24.227133 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:26.187145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:26.187103 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:26.395085 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:24:26.395041 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:24:32.222154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:32.222113 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:34.226059 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:34.226017 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:36.187552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:36.187495 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:42.222187 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:42.222155 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:24:44.324564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:44.324523 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:44.324995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:44.324701 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:24:46.188136 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:46.188095 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:47.421201 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.421165 4988 generic.go:356] "Generic (PLEG): container finished" podID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerID="3be3fa218b82a436590bf579774c986db0d42a665581f8f589b4b7cca1b63202" exitCode=137 Oct 30 21:24:47.421654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.421238 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" event={"ID":"5f0ac17f-620d-4da9-803c-3208b2cec95c","Type":"ContainerDied","Data":"3be3fa218b82a436590bf579774c986db0d42a665581f8f589b4b7cca1b63202"} Oct 30 21:24:47.663466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.663438 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:24:47.778523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.778439 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf\") pod \"5f0ac17f-620d-4da9-803c-3208b2cec95c\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " Oct 30 21:24:47.778523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.778475 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle\") pod \"5f0ac17f-620d-4da9-803c-3208b2cec95c\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " Oct 30 21:24:47.778523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.778509 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls\") pod \"5f0ac17f-620d-4da9-803c-3208b2cec95c\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " Oct 30 21:24:47.778805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.778540 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7\" (UniqueName: \"kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7\") pod \"5f0ac17f-620d-4da9-803c-3208b2cec95c\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " Oct 30 21:24:47.778805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.778598 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage\") pod \"5f0ac17f-620d-4da9-803c-3208b2cec95c\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " Oct 30 21:24:47.778805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.778657 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key\") pod \"5f0ac17f-620d-4da9-803c-3208b2cec95c\" (UID: \"5f0ac17f-620d-4da9-803c-3208b2cec95c\") " Oct 30 21:24:47.779075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.779047 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf" pod "5f0ac17f-620d-4da9-803c-3208b2cec95c" (UID: "5f0ac17f-620d-4da9-803c-3208b2cec95c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:47.779075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.779059 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage" pod "5f0ac17f-620d-4da9-803c-3208b2cec95c" (UID: "5f0ac17f-620d-4da9-803c-3208b2cec95c"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:47.779230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.779062 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle" pod "5f0ac17f-620d-4da9-803c-3208b2cec95c" (UID: "5f0ac17f-620d-4da9-803c-3208b2cec95c"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:47.780742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.780708 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls" pod "5f0ac17f-620d-4da9-803c-3208b2cec95c" (UID: "5f0ac17f-620d-4da9-803c-3208b2cec95c"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:47.780834 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.780731 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key" pod "5f0ac17f-620d-4da9-803c-3208b2cec95c" (UID: "5f0ac17f-620d-4da9-803c-3208b2cec95c"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:47.780834 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.780779 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7" pod "5f0ac17f-620d-4da9-803c-3208b2cec95c" (UID: "5f0ac17f-620d-4da9-803c-3208b2cec95c"). InnerVolumeSpecName "kube-api-access-4mmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:47.880144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.880105 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.880144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.880138 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.880144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.880151 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.880390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.880162 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mmm7\" (UniqueName: \"kubernetes.io/projected/5f0ac17f-620d-4da9-803c-3208b2cec95c-kube-api-access-4mmm7\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.880390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.880173 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5f0ac17f-620d-4da9-803c-3208b2cec95c-tempo-tmp-storage\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.880390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:47.880182 4988 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/5f0ac17f-620d-4da9-803c-3208b2cec95c-storage-gcs-key\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:48.427732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:48.427700 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" Oct 30 21:24:48.427732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:48.427721 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj" event={"ID":"5f0ac17f-620d-4da9-803c-3208b2cec95c","Type":"ContainerDied","Data":"a7bfa3abed6aa14838b5e7d4cea3b3ec6e6ab004f6745bd057dca03529c0d709"} Oct 30 21:24:48.428196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:48.427789 4988 scope.go:122] "RemoveContainer" containerID="3be3fa218b82a436590bf579774c986db0d42a665581f8f589b4b7cca1b63202" Oct 30 21:24:48.595956 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:48.595916 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj"] Oct 30 21:24:48.645668 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:48.645633 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7bb787577-d76lj"] Oct 30 21:24:50.015884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:50.015851 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" path="/var/lib/kubelet/pods/5f0ac17f-620d-4da9-803c-3208b2cec95c/volumes" Oct 30 21:24:56.187242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:56.187160 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:24:56.395819 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:24:56.395780 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:24:58.899142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:58.899097 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:59.094972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:24:59.094934 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:00.765286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:00.765245 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:00.765755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:00.765487 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c968c75b-4054-404c-9215-70a2f1ab8c26" containerName="otc-container" containerID="cri-o://304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5" gracePeriod=30 Oct 30 21:25:01.016490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.016397 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:01.089338 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.089303 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:01.197696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.197651 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal\") pod \"c968c75b-4054-404c-9215-70a2f1ab8c26\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " Oct 30 21:25:01.197909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.197735 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs\") pod \"c968c75b-4054-404c-9215-70a2f1ab8c26\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " Oct 30 21:25:01.197909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.197805 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs\") pod \"c968c75b-4054-404c-9215-70a2f1ab8c26\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " Oct 30 21:25:01.197909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.197894 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk\" (UniqueName: \"kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk\") pod \"c968c75b-4054-404c-9215-70a2f1ab8c26\" (UID: \"c968c75b-4054-404c-9215-70a2f1ab8c26\") " Oct 30 21:25:01.198132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.198106 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal" pod "c968c75b-4054-404c-9215-70a2f1ab8c26" (UID: "c968c75b-4054-404c-9215-70a2f1ab8c26"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:01.200078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.200038 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs" pod "c968c75b-4054-404c-9215-70a2f1ab8c26" (UID: "c968c75b-4054-404c-9215-70a2f1ab8c26"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:01.200219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.200082 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk" pod "c968c75b-4054-404c-9215-70a2f1ab8c26" (UID: "c968c75b-4054-404c-9215-70a2f1ab8c26"). InnerVolumeSpecName "kube-api-access-d5dnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:01.200219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.200180 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs" pod "c968c75b-4054-404c-9215-70a2f1ab8c26" (UID: "c968c75b-4054-404c-9215-70a2f1ab8c26"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:01.235766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.235728 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:01.299173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.299079 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dnk\" (UniqueName: \"kubernetes.io/projected/c968c75b-4054-404c-9215-70a2f1ab8c26-kube-api-access-d5dnk\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:01.299173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.299116 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c968c75b-4054-404c-9215-70a2f1ab8c26-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:01.299173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.299125 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:01.299173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.299133 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968c75b-4054-404c-9215-70a2f1ab8c26-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:01.488159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.488120 4988 generic.go:356] "Generic (PLEG): container finished" podID="c968c75b-4054-404c-9215-70a2f1ab8c26" containerID="304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5" exitCode=0 Oct 30 21:25:01.488336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.488170 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:01.488336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.488183 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c968c75b-4054-404c-9215-70a2f1ab8c26","Type":"ContainerDied","Data":"304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5"} Oct 30 21:25:01.488336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.488238 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c968c75b-4054-404c-9215-70a2f1ab8c26","Type":"ContainerDied","Data":"604d4fe4fad61b08ff94b016d3a26983ad1ebc5983b5bb25849264235b3ca8c5"} Oct 30 21:25:01.488336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.488269 4988 scope.go:122] "RemoveContainer" containerID="304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5" Oct 30 21:25:01.497335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.497314 4988 scope.go:122] "RemoveContainer" containerID="304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5" Oct 30 21:25:01.497725 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:25:01.497705 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5\": container with ID starting with 304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5 not found: ID does not exist" containerID="304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5" Oct 30 21:25:01.497782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.497735 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5"} err="failed to get container status \"304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5\": rpc error: code = NotFound desc = could not find container \"304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5\": container with ID starting with 304433d300bca4a857c71e0b120235c113e379c50e54f337f1d20bb40c9941f5 not found: ID does not exist" Oct 30 21:25:01.652512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.652474 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:01.658530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.658500 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:01.736739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.736705 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:01.737051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737039 4988 cpu_manager.go:414] "containerMap: removing container" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" Oct 30 21:25:01.737104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737052 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" Oct 30 21:25:01.737104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737065 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c968c75b-4054-404c-9215-70a2f1ab8c26" containerName="otc-container" Oct 30 21:25:01.737104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737070 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c968c75b-4054-404c-9215-70a2f1ab8c26" containerName="otc-container" Oct 30 21:25:01.737104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737082 4988 cpu_manager.go:414] "containerMap: removing container" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway" Oct 30 21:25:01.737104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737087 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway" Oct 30 21:25:01.737104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737105 4988 cpu_manager.go:414] "containerMap: removing container" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway-opa" Oct 30 21:25:01.737278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737110 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway-opa" Oct 30 21:25:01.737278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737176 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway" Oct 30 21:25:01.737278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737186 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="429687ca-16c3-4e87-8914-84f84ee3f22d" containerName="tempo-gateway-opa" Oct 30 21:25:01.737278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737193 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0ac17f-620d-4da9-803c-3208b2cec95c" containerName="tempo" Oct 30 21:25:01.737278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.737198 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c968c75b-4054-404c-9215-70a2f1ab8c26" containerName="otc-container" Oct 30 21:25:01.741477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.741457 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:01.794345 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.794309 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:25:01.794345 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.794309 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:25:01.794790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.794401 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 21:25:01.794790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.794440 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-nfqbf\"" Oct 30 21:25:01.799971 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.799927 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:01.905195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.905103 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:01.905195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.905180 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6gc\" (UniqueName: \"kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:01.905405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.905215 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:01.905405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:01.905238 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.006638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.006605 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.006789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.006653 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.006789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.006722 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.006789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.006780 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6gc\" (UniqueName: \"kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.007383 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.007359 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.009130 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.009109 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.009216 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.009165 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.016926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.016890 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c968c75b-4054-404c-9215-70a2f1ab8c26" path="/var/lib/kubelet/pods/c968c75b-4054-404c-9215-70a2f1ab8c26/volumes" Oct 30 21:25:02.068663 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.068631 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6gc\" (UniqueName: \"kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc\") pod \"data-science-collector-collector-1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.353654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.353564 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:02.766371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.766334 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:02.770383 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:25:02.770345 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60081658_7edf_4d17_8df0_7bee8b8ee2b1.slice/crio-b67f0e06e5d5753b51990d5253af61f2f5f5b07fa27386d4f6690ecec3ad5e3a WatchSource:0}: Error finding container b67f0e06e5d5753b51990d5253af61f2f5f5b07fa27386d4f6690ecec3ad5e3a: Status 404 returned error can't find the container with id b67f0e06e5d5753b51990d5253af61f2f5f5b07fa27386d4f6690ecec3ad5e3a Oct 30 21:25:02.772538 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.772504 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:02.772620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:02.772570 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:03.503283 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:03.503241 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"60081658-7edf-4d17-8df0-7bee8b8ee2b1","Type":"ContainerStarted","Data":"a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0"} Oct 30 21:25:03.503830 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:03.503339 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"60081658-7edf-4d17-8df0-7bee8b8ee2b1","Type":"ContainerStarted","Data":"b67f0e06e5d5753b51990d5253af61f2f5f5b07fa27386d4f6690ecec3ad5e3a"} Oct 30 21:25:03.622355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:03.622297 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.622285715 podStartE2EDuration="2.622285715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:03.621546702 +0000 UTC m=+2931.729042615" watchObservedRunningTime="2026-10-30 21:25:03.622285715 +0000 UTC m=+2931.729781626" Oct 30 21:25:18.274476 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:18.274438 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:26.393959 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:25:26.393916 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:25:37.942037 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:37.941985 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:37.942587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:37.942207 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="60081658-7edf-4d17-8df0-7bee8b8ee2b1" containerName="otc-container" containerID="cri-o://a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0" gracePeriod=30 Oct 30 21:25:37.962386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:37.962352 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:38.106850 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.106815 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:38.197244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.197174 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:38.320308 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.320276 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal\") pod \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " Oct 30 21:25:38.320516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.320390 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs\") pod \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " Oct 30 21:25:38.320516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.320472 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs\") pod \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " Oct 30 21:25:38.320642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.320535 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc\" (UniqueName: \"kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc\") pod \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\" (UID: \"60081658-7edf-4d17-8df0-7bee8b8ee2b1\") " Oct 30 21:25:38.320784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.320729 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal" pod "60081658-7edf-4d17-8df0-7bee8b8ee2b1" (UID: "60081658-7edf-4d17-8df0-7bee8b8ee2b1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:38.322263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.322147 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:38.323089 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.323058 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs" pod "60081658-7edf-4d17-8df0-7bee8b8ee2b1" (UID: "60081658-7edf-4d17-8df0-7bee8b8ee2b1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:38.323439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.323386 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc" pod "60081658-7edf-4d17-8df0-7bee8b8ee2b1" (UID: "60081658-7edf-4d17-8df0-7bee8b8ee2b1"). InnerVolumeSpecName "kube-api-access-2z6gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:38.323555 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.323448 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs" pod "60081658-7edf-4d17-8df0-7bee8b8ee2b1" (UID: "60081658-7edf-4d17-8df0-7bee8b8ee2b1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:38.422003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.421962 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:38.422003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.421997 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6gc\" (UniqueName: \"kubernetes.io/projected/60081658-7edf-4d17-8df0-7bee8b8ee2b1-kube-api-access-2z6gc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:38.422003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.422007 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/60081658-7edf-4d17-8df0-7bee8b8ee2b1-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:38.422249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.422016 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/60081658-7edf-4d17-8df0-7bee8b8ee2b1-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:38.660298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.660261 4988 generic.go:356] "Generic (PLEG): container finished" podID="60081658-7edf-4d17-8df0-7bee8b8ee2b1" containerID="a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0" exitCode=0 Oct 30 21:25:38.660505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.660306 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:38.660505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.660325 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"60081658-7edf-4d17-8df0-7bee8b8ee2b1","Type":"ContainerDied","Data":"a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0"} Oct 30 21:25:38.660505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.660367 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"60081658-7edf-4d17-8df0-7bee8b8ee2b1","Type":"ContainerDied","Data":"b67f0e06e5d5753b51990d5253af61f2f5f5b07fa27386d4f6690ecec3ad5e3a"} Oct 30 21:25:38.660505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.660391 4988 scope.go:122] "RemoveContainer" containerID="a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0" Oct 30 21:25:38.669546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.669526 4988 scope.go:122] "RemoveContainer" containerID="a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0" Oct 30 21:25:38.669815 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:25:38.669796 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0\": container with ID starting with a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0 not found: ID does not exist" containerID="a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0" Oct 30 21:25:38.669862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.669825 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0"} err="failed to get container status \"a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0\": rpc error: code = NotFound desc = could not find container \"a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0\": container with ID starting with a9d44cee857a64641abe03df9503f07b0d1cd2aded76725a5e05111639ceeba0 not found: ID does not exist" Oct 30 21:25:38.831016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.830959 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:38.835060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.835009 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:38.843953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.843919 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:38.857836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.857800 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:25:38.858097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.858071 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerName="tempo" containerID="cri-o://1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60" gracePeriod=30 Oct 30 21:25:38.928084 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.927998 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw"] Oct 30 21:25:38.928316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.928290 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" containerID="cri-o://27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468" gracePeriod=30 Oct 30 21:25:38.932691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.932654 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26"] Oct 30 21:25:38.933241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.932914 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" containerID="cri-o://3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50" gracePeriod=30 Oct 30 21:25:38.934785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.934754 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2"] Oct 30 21:25:38.935034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.935008 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" podUID="0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" containerName="tempo" containerID="cri-o://0c7f36666225fde4b6097659faa3527f489fe25a4d39ef3ac689424db3336aff" gracePeriod=30 Oct 30 21:25:38.992710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:38.992676 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:39.158918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.158886 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:39.666768 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.666728 4988 generic.go:356] "Generic (PLEG): container finished" podID="0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" containerID="0c7f36666225fde4b6097659faa3527f489fe25a4d39ef3ac689424db3336aff" exitCode=0 Oct 30 21:25:39.666944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.666780 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" event={"ID":"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94","Type":"ContainerDied","Data":"0c7f36666225fde4b6097659faa3527f489fe25a4d39ef3ac689424db3336aff"} Oct 30 21:25:39.793429 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.793390 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:25:39.938606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.938570 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls\") pod \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " Oct 30 21:25:39.938824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.938648 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf\") pod \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " Oct 30 21:25:39.938824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.938677 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage\") pod \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " Oct 30 21:25:39.938824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.938755 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle\") pod \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " Oct 30 21:25:39.938824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.938798 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns\" (UniqueName: \"kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns\") pod \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\" (UID: \"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94\") " Oct 30 21:25:39.939091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.938994 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage" pod "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" (UID: "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:39.939164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.939096 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-tmp-storage\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:39.939225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.939183 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf" pod "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" (UID: "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:39.939225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.939193 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle" pod "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" (UID: "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:39.940961 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.940929 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls" pod "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" (UID: "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:39.941068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:39.940982 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns" pod "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" (UID: "0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94"). InnerVolumeSpecName "kube-api-access-hgdns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:40.016234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.016198 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60081658-7edf-4d17-8df0-7bee8b8ee2b1" path="/var/lib/kubelet/pods/60081658-7edf-4d17-8df0-7bee8b8ee2b1/volumes" Oct 30 21:25:40.039975 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.039936 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.039975 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.039972 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.040189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.039989 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgdns\" (UniqueName: \"kubernetes.io/projected/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-kube-api-access-hgdns\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.040189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.040003 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:40.674029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.674002 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" Oct 30 21:25:40.674209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.674042 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2" event={"ID":"0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94","Type":"ContainerDied","Data":"a5f2b925dfb4ffe6ff27d5ca6dbc83aa8ad98123c45e76a78ba05431f32faa38"} Oct 30 21:25:40.674209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.674098 4988 scope.go:122] "RemoveContainer" containerID="0c7f36666225fde4b6097659faa3527f489fe25a4d39ef3ac689424db3336aff" Oct 30 21:25:40.838430 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.838375 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2"] Oct 30 21:25:40.888112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:40.888075 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-b8c4ff99-99bx2"] Oct 30 21:25:41.203089 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.203058 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:25:41.351933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.351842 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc\" (UniqueName: \"kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc\") pod \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " Oct 30 21:25:41.351933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.351894 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls\") pod \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " Oct 30 21:25:41.352395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.351940 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle\") pod \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " Oct 30 21:25:41.352395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.352043 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") pod \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " Oct 30 21:25:41.352395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.352197 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf\") pod \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " Oct 30 21:25:41.352395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.352229 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key\") pod \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\" (UID: \"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9\") " Oct 30 21:25:41.352644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.352497 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle" pod "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" (UID: "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:41.352644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.352626 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf" pod "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" (UID: "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:41.354404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.354370 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc" pod "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" (UID: "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9"). InnerVolumeSpecName "kube-api-access-7q2tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:41.354404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.354394 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls" pod "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" (UID: "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:41.354573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.354448 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key" pod "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" (UID: "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:41.355001 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.354982 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62" pod "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" (UID: "d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9"). InnerVolumeSpecName "pvc-8413c93a-423e-4f5d-b17d-1465d72f5969". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:25:41.453634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.453589 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:41.453827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.453659 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:25:41.453827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.453675 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:41.453827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.453691 4988 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-storage-gcs-key\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:41.453827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.453706 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2tc\" (UniqueName: \"kubernetes.io/projected/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-kube-api-access-7q2tc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:41.453827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.453720 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:25:41.462461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.462436 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-8413c93a-423e-4f5d-b17d-1465d72f5969" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:25:41.554874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.554828 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-8413c93a-423e-4f5d-b17d-1465d72f5969\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-01e1ef8c3cf2a6b62\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-113b31927c29ab22b082280e8c62c14f67a0caad66f0a4302e0e79216886b34c\"" Oct 30 21:25:41.681388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.681343 4988 generic.go:356] "Generic (PLEG): container finished" podID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerID="1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60" exitCode=0 Oct 30 21:25:41.681614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.681391 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 30 21:25:41.681614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.681399 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9","Type":"ContainerDied","Data":"1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60"} Oct 30 21:25:41.681614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.681477 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9","Type":"ContainerDied","Data":"a79689a965926f624b9d479ca598ba4c927f03167513feb6a7d8a78aeaf67b7d"} Oct 30 21:25:41.681614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.681513 4988 scope.go:122] "RemoveContainer" containerID="1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60" Oct 30 21:25:41.691344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.691325 4988 scope.go:122] "RemoveContainer" containerID="1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60" Oct 30 21:25:41.691746 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:25:41.691724 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60\": container with ID starting with 1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60 not found: ID does not exist" containerID="1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60" Oct 30 21:25:41.691815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.691755 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60"} err="failed to get container status \"1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60\": rpc error: code = NotFound desc = could not find container \"1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60\": container with ID starting with 1f973a808f994fee79e179de64af7b0d667f6f5be9994f9d688a715e635c8a60 not found: ID does not exist" Oct 30 21:25:41.845752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.845712 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:25:41.895438 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:41.895390 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 30 21:25:42.016995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:42.016915 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" path="/var/lib/kubelet/pods/0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94/volumes" Oct 30 21:25:42.017343 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:42.017329 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" path="/var/lib/kubelet/pods/d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9/volumes" Oct 30 21:25:43.383440 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:43.383378 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:46.188458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:46.188395 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:46.188874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:46.188462 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:46.725838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:46.725795 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:46.923134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:46.923094 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:47.068819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:47.068722 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:48.117335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:48.117297 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:48.889145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:48.889104 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:49.071699 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.071662 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:49.072143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072121 4988 cpu_manager.go:414] "containerMap: removing container" podUID="0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" containerName="tempo" Oct 30 21:25:49.072143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072143 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" containerName="tempo" Oct 30 21:25:49.072321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072164 4988 cpu_manager.go:414] "containerMap: removing container" podUID="60081658-7edf-4d17-8df0-7bee8b8ee2b1" containerName="otc-container" Oct 30 21:25:49.072321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072172 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60081658-7edf-4d17-8df0-7bee8b8ee2b1" containerName="otc-container" Oct 30 21:25:49.072321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072208 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerName="tempo" Oct 30 21:25:49.072321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072216 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerName="tempo" Oct 30 21:25:49.072321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072311 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c091f0d-d1ee-49b1-b9bf-f2c95aa90a94" containerName="tempo" Oct 30 21:25:49.072583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072323 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="60081658-7edf-4d17-8df0-7bee8b8ee2b1" containerName="otc-container" Oct 30 21:25:49.072583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072337 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a6a590-0e64-4fa1-b0f1-a8ef8cee11d9" containerName="tempo" Oct 30 21:25:49.072583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.072520 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:49.076840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.076823 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.134604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.134552 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:25:49.136161 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.136130 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:49.184265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.184181 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 21:25:49.218643 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.218601 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9gv\" (UniqueName: \"kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.218799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.218646 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.218799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.218670 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.218888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.218800 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.235884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.235849 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8qdwd\"" Oct 30 21:25:49.236585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.236568 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:25:49.245948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.245909 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:49.319262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.319220 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9gv\" (UniqueName: \"kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.319262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.319259 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.319542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.319279 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.319542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.319316 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.320044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.320017 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.321759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.321738 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.321851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.321789 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.380749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.380707 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9gv\" (UniqueName: \"kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv\") pod \"data-science-collector-collector-1\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.390305 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.390269 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:25:49.783231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.783141 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:25:49.787317 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:25:49.787286 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e768ca_7771_47ec_b47e_b395a7a4dcde.slice/crio-6c82847b28eca19b7a524a54394301c07f2abbddea160a20e723fb8b69d473f7 WatchSource:0}: Error finding container 6c82847b28eca19b7a524a54394301c07f2abbddea160a20e723fb8b69d473f7: Status 404 returned error can't find the container with id 6c82847b28eca19b7a524a54394301c07f2abbddea160a20e723fb8b69d473f7 Oct 30 21:25:49.789492 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.789458 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:49.789592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:49.789544 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:50.423132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.423092 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:25:50.426823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.426797 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.436422 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.436372 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:50.480594 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.480558 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 21:25:50.480594 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.480557 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:25:50.480594 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.480559 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:25:50.480873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.480572 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 21:25:50.480873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.480637 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 21:25:50.530204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.530150 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 21:25:50.530204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.530169 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 21:25:50.530204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.530188 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-k7c72\"" Oct 30 21:25:50.531925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.531897 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:25:50.634106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634066 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2cb\" (UniqueName: \"kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634118 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634181 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634201 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634219 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634251 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634336 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.634592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.634403 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.642375 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.642340 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:50.728215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.728121 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7e768ca-7771-47ec-b47e-b395a7a4dcde","Type":"ContainerStarted","Data":"c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552"} Oct 30 21:25:50.728215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.728196 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7e768ca-7771-47ec-b47e-b395a7a4dcde","Type":"ContainerStarted","Data":"6c82847b28eca19b7a524a54394301c07f2abbddea160a20e723fb8b69d473f7"} Oct 30 21:25:50.735792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735755 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2cb\" (UniqueName: \"kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735805 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735830 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735849 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735864 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735892 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735909 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.735952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.735928 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.736677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.736644 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.736797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.736705 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.736890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.736868 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.736928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.736868 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.738509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.738483 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.738802 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.738784 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.738875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.738838 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.797045 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.797006 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2cb\" (UniqueName: \"kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:50.846984 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:50.846925 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.846909452 podStartE2EDuration="1.846909452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:50.845481579 +0000 UTC m=+2978.952977492" watchObservedRunningTime="2026-10-30 21:25:50.846909452 +0000 UTC m=+2978.954405362" Oct 30 21:25:52.251102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.251014 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.354468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.354400 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.453182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.453142 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.453373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.453281 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.505087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.505001 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.643802 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.643763 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.709130 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.709103 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:52.844657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:52.844559 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:53.331061 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:25:53.331017 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd011eab9_c0d8_403e_85bc_96a5276c1fa9.slice/crio-c116235466519e45158961420a10f454651c688243055dafb6ede811a30bd6cd WatchSource:0}: Error finding container c116235466519e45158961420a10f454651c688243055dafb6ede811a30bd6cd: Status 404 returned error can't find the container with id c116235466519e45158961420a10f454651c688243055dafb6ede811a30bd6cd Oct 30 21:25:53.377458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.377426 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:25:53.744889 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.744849 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerStarted","Data":"b8205b11a6405bcb11c6e62feb8895f61c946a223db2dd6e2604d4be985ae477"} Oct 30 21:25:53.745064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.744916 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerStarted","Data":"208d0977763d09b121b2996752e392eb25af92863bae0c33b0aaf9222ca8e738"} Oct 30 21:25:53.745064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.744941 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerStarted","Data":"55a0dbaf2a8821f7489ee2389ae0aca365f7e6e5a715f9a1ded322d14ae23366"} Oct 30 21:25:53.745064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.744961 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerStarted","Data":"c116235466519e45158961420a10f454651c688243055dafb6ede811a30bd6cd"} Oct 30 21:25:53.745064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.744987 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:53.747792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.747766 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:53.869970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:53.869918 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=3.869903938 podStartE2EDuration="3.869903938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:53.868305022 +0000 UTC m=+2981.975800933" watchObservedRunningTime="2026-10-30 21:25:53.869903938 +0000 UTC m=+2981.977399849" Oct 30 21:25:54.749712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:54.749676 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:54.750129 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:54.749733 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:54.757287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:54.757260 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:25:56.187457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:56.187372 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:56.187895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:56.187864 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:56.395390 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:25:56.395349 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:25:59.220229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:59.220165 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:59.378366 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:25:59.378330 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:00.192371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.192308 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:26:00.321534 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.321495 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key\") pod \"9ecefc32-81b3-4831-a88c-a6bef976d270\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " Oct 30 21:26:00.322036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.321553 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle\") pod \"9ecefc32-81b3-4831-a88c-a6bef976d270\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " Oct 30 21:26:00.322036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.321573 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54\" (UniqueName: \"kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54\") pod \"9ecefc32-81b3-4831-a88c-a6bef976d270\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " Oct 30 21:26:00.322036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.321617 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf\") pod \"9ecefc32-81b3-4831-a88c-a6bef976d270\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " Oct 30 21:26:00.322036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.321669 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage\") pod \"9ecefc32-81b3-4831-a88c-a6bef976d270\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " Oct 30 21:26:00.322036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.321711 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls\") pod \"9ecefc32-81b3-4831-a88c-a6bef976d270\" (UID: \"9ecefc32-81b3-4831-a88c-a6bef976d270\") " Oct 30 21:26:00.322313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.322087 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle" pod "9ecefc32-81b3-4831-a88c-a6bef976d270" (UID: "9ecefc32-81b3-4831-a88c-a6bef976d270"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:00.322313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.322126 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage" pod "9ecefc32-81b3-4831-a88c-a6bef976d270" (UID: "9ecefc32-81b3-4831-a88c-a6bef976d270"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:00.322313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.322148 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf" pod "9ecefc32-81b3-4831-a88c-a6bef976d270" (UID: "9ecefc32-81b3-4831-a88c-a6bef976d270"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:00.323958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.323929 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key" pod "9ecefc32-81b3-4831-a88c-a6bef976d270" (UID: "9ecefc32-81b3-4831-a88c-a6bef976d270"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:00.323958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.323944 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54" pod "9ecefc32-81b3-4831-a88c-a6bef976d270" (UID: "9ecefc32-81b3-4831-a88c-a6bef976d270"). InnerVolumeSpecName "kube-api-access-qmq54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:00.324099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.323959 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls" pod "9ecefc32-81b3-4831-a88c-a6bef976d270" (UID: "9ecefc32-81b3-4831-a88c-a6bef976d270"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:00.422633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.422573 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.422633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.422626 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmq54\" (UniqueName: \"kubernetes.io/projected/9ecefc32-81b3-4831-a88c-a6bef976d270-kube-api-access-qmq54\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.422633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.422640 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.422633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.422649 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-tmp-storage\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.422926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.422658 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.422926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.422667 4988 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/9ecefc32-81b3-4831-a88c-a6bef976d270-storage-gcs-key\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.779495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.779457 4988 generic.go:356] "Generic (PLEG): container finished" podID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerID="3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50" exitCode=0 Oct 30 21:26:00.779730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.779504 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" event={"ID":"9ecefc32-81b3-4831-a88c-a6bef976d270","Type":"ContainerDied","Data":"3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50"} Oct 30 21:26:00.779730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.779511 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" Oct 30 21:26:00.779730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.779546 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26" event={"ID":"9ecefc32-81b3-4831-a88c-a6bef976d270","Type":"ContainerDied","Data":"6cefc057603587d98ca3c4ed3fcb1564d4526ee2b0b640c1d5ca867103bc0cfb"} Oct 30 21:26:00.779730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.779569 4988 scope.go:122] "RemoveContainer" containerID="3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50" Oct 30 21:26:00.791218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.791193 4988 scope.go:122] "RemoveContainer" containerID="3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50" Oct 30 21:26:00.791600 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:26:00.791563 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50\": container with ID starting with 3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50 not found: ID does not exist" containerID="3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50" Oct 30 21:26:00.791695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.791612 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50"} err="failed to get container status \"3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50\": rpc error: code = NotFound desc = could not find container \"3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50\": container with ID starting with 3035d20b626ceec3b718cb9d0a9d1bd372d174dccfe55752bb76d554a7d49f50 not found: ID does not exist" Oct 30 21:26:00.944200 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.944165 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26"] Oct 30 21:26:00.994380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:00.994343 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-557bcbd889-rsj26"] Oct 30 21:26:02.016564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:02.016526 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" path="/var/lib/kubelet/pods/9ecefc32-81b3-4831-a88c-a6bef976d270/volumes" Oct 30 21:26:06.186983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:06.186943 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:26:06.187386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:06.187066 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:26:09.129254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.129216 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:26:09.204024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.203930 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key\") pod \"fd64e775-f161-4439-bf7b-5348c458676b\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " Oct 30 21:26:09.204024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.203995 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf\") pod \"fd64e775-f161-4439-bf7b-5348c458676b\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " Oct 30 21:26:09.204273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204038 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage\") pod \"fd64e775-f161-4439-bf7b-5348c458676b\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " Oct 30 21:26:09.204273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204074 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg\" (UniqueName: \"kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg\") pod \"fd64e775-f161-4439-bf7b-5348c458676b\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " Oct 30 21:26:09.204273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204107 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls\") pod \"fd64e775-f161-4439-bf7b-5348c458676b\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " Oct 30 21:26:09.204273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204130 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle\") pod \"fd64e775-f161-4439-bf7b-5348c458676b\" (UID: \"fd64e775-f161-4439-bf7b-5348c458676b\") " Oct 30 21:26:09.204532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204493 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage" pod "fd64e775-f161-4439-bf7b-5348c458676b" (UID: "fd64e775-f161-4439-bf7b-5348c458676b"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:09.204588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204557 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf" pod "fd64e775-f161-4439-bf7b-5348c458676b" (UID: "fd64e775-f161-4439-bf7b-5348c458676b"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:09.204633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.204580 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle" pod "fd64e775-f161-4439-bf7b-5348c458676b" (UID: "fd64e775-f161-4439-bf7b-5348c458676b"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:09.206219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.206187 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls" pod "fd64e775-f161-4439-bf7b-5348c458676b" (UID: "fd64e775-f161-4439-bf7b-5348c458676b"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:09.206219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.206211 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key" pod "fd64e775-f161-4439-bf7b-5348c458676b" (UID: "fd64e775-f161-4439-bf7b-5348c458676b"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:09.206379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.206237 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg" pod "fd64e775-f161-4439-bf7b-5348c458676b" (UID: "fd64e775-f161-4439-bf7b-5348c458676b"). InnerVolumeSpecName "kube-api-access-lh7dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:09.305527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.305487 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7dg\" (UniqueName: \"kubernetes.io/projected/fd64e775-f161-4439-bf7b-5348c458676b-kube-api-access-lh7dg\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.305527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.305529 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.305784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.305544 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.305784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.305559 4988 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/fd64e775-f161-4439-bf7b-5348c458676b-storage-gcs-key\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.305784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.305573 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/fd64e775-f161-4439-bf7b-5348c458676b-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.305784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.305589 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/fd64e775-f161-4439-bf7b-5348c458676b-tempo-tmp-storage\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:09.823375 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.823339 4988 generic.go:356] "Generic (PLEG): container finished" podID="fd64e775-f161-4439-bf7b-5348c458676b" containerID="27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468" exitCode=137 Oct 30 21:26:09.823599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.823384 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" Oct 30 21:26:09.823599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.823445 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" event={"ID":"fd64e775-f161-4439-bf7b-5348c458676b","Type":"ContainerDied","Data":"27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468"} Oct 30 21:26:09.823599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.823504 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw" event={"ID":"fd64e775-f161-4439-bf7b-5348c458676b","Type":"ContainerDied","Data":"a26fa20c54ed6db27135a0f93e9bf9c578e3b360080b22a9f04b507a3225926e"} Oct 30 21:26:09.823599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.823534 4988 scope.go:122] "RemoveContainer" containerID="27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468" Oct 30 21:26:09.833749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.833729 4988 scope.go:122] "RemoveContainer" containerID="27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468" Oct 30 21:26:09.834130 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:26:09.834110 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468\": container with ID starting with 27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468 not found: ID does not exist" containerID="27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468" Oct 30 21:26:09.834183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.834140 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468"} err="failed to get container status \"27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468\": rpc error: code = NotFound desc = could not find container \"27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468\": container with ID starting with 27bdb39123ac35f21be64f8aff988d3ec5e82713129c39c5491d59fcb7289468 not found: ID does not exist" Oct 30 21:26:09.987730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:09.987689 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw"] Oct 30 21:26:10.037223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:10.037188 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-59776c68c7-bwqtw"] Oct 30 21:26:12.022725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:12.022694 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd64e775-f161-4439-bf7b-5348c458676b" path="/var/lib/kubelet/pods/fd64e775-f161-4439-bf7b-5348c458676b/volumes" Oct 30 21:26:12.307083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:12.306940 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:26:12.320696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:12.315776 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:26:12.994559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:12.994521 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:26:12.994748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:12.994726 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a7e768ca-7771-47ec-b47e-b395a7a4dcde" containerName="otc-container" containerID="cri-o://c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552" gracePeriod=30 Oct 30 21:26:13.012118 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.012084 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:13.160970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.160934 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:13.258738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.258675 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:26:13.341995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.341950 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal\") pod \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " Oct 30 21:26:13.342184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.342008 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs\") pod \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " Oct 30 21:26:13.342184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.342038 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs\") pod \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " Oct 30 21:26:13.342184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.342131 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv\" (UniqueName: \"kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv\") pod \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\" (UID: \"a7e768ca-7771-47ec-b47e-b395a7a4dcde\") " Oct 30 21:26:13.342476 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.342450 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal" pod "a7e768ca-7771-47ec-b47e-b395a7a4dcde" (UID: "a7e768ca-7771-47ec-b47e-b395a7a4dcde"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:13.344219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.344190 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs" pod "a7e768ca-7771-47ec-b47e-b395a7a4dcde" (UID: "a7e768ca-7771-47ec-b47e-b395a7a4dcde"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:13.344661 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.344631 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs" pod "a7e768ca-7771-47ec-b47e-b395a7a4dcde" (UID: "a7e768ca-7771-47ec-b47e-b395a7a4dcde"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:13.344661 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.344641 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv" pod "a7e768ca-7771-47ec-b47e-b395a7a4dcde" (UID: "a7e768ca-7771-47ec-b47e-b395a7a4dcde"). InnerVolumeSpecName "kube-api-access-2b9gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:13.347748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.347715 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:13.442766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.442729 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:13.442766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.442763 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:13.442766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.442772 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7e768ca-7771-47ec-b47e-b395a7a4dcde-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:13.443008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.442781 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9gv\" (UniqueName: \"kubernetes.io/projected/a7e768ca-7771-47ec-b47e-b395a7a4dcde-kube-api-access-2b9gv\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:13.712661 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.712621 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:26:13.712905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.712875 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo" containerID="cri-o://55a0dbaf2a8821f7489ee2389ae0aca365f7e6e5a715f9a1ded322d14ae23366" gracePeriod=30 Oct 30 21:26:13.713021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.712930 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway-opa" containerID="cri-o://b8205b11a6405bcb11c6e62feb8895f61c946a223db2dd6e2604d4be985ae477" gracePeriod=30 Oct 30 21:26:13.713087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.712939 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway" containerID="cri-o://208d0977763d09b121b2996752e392eb25af92863bae0c33b0aaf9222ca8e738" gracePeriod=30 Oct 30 21:26:13.719670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.719625 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:26:13.728995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.728954 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:13.746170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.746130 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.98:8083/ready\": dial tcp 10.132.0.98:8083: connect: connection refused" Oct 30 21:26:13.848478 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.848446 4988 generic.go:356] "Generic (PLEG): container finished" podID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerID="b8205b11a6405bcb11c6e62feb8895f61c946a223db2dd6e2604d4be985ae477" exitCode=0 Oct 30 21:26:13.848607 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.848522 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerDied","Data":"b8205b11a6405bcb11c6e62feb8895f61c946a223db2dd6e2604d4be985ae477"} Oct 30 21:26:13.850081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.850053 4988 generic.go:356] "Generic (PLEG): container finished" podID="a7e768ca-7771-47ec-b47e-b395a7a4dcde" containerID="c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552" exitCode=0 Oct 30 21:26:13.850195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.850105 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7e768ca-7771-47ec-b47e-b395a7a4dcde","Type":"ContainerDied","Data":"c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552"} Oct 30 21:26:13.850195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.850156 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7e768ca-7771-47ec-b47e-b395a7a4dcde","Type":"ContainerDied","Data":"6c82847b28eca19b7a524a54394301c07f2abbddea160a20e723fb8b69d473f7"} Oct 30 21:26:13.850195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.850186 4988 scope.go:122] "RemoveContainer" containerID="c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552" Oct 30 21:26:13.850465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.850184 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:26:13.925479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.925449 4988 scope.go:122] "RemoveContainer" containerID="c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552" Oct 30 21:26:13.925824 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:26:13.925803 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552\": container with ID starting with c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552 not found: ID does not exist" containerID="c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552" Oct 30 21:26:13.925917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.925843 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552"} err="failed to get container status \"c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552\": rpc error: code = NotFound desc = could not find container \"c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552\": container with ID starting with c57bdda6c675dcbb65b3e0d8bfc4a81b81697800cba8db7f1ff1b1375dd6a552 not found: ID does not exist" Oct 30 21:26:13.961309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:13.961275 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:14.016722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.016692 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:26:14.019316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.019288 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:26:14.137248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.137218 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:14.754959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.754925 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:26:14.857212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.857176 4988 generic.go:356] "Generic (PLEG): container finished" podID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerID="208d0977763d09b121b2996752e392eb25af92863bae0c33b0aaf9222ca8e738" exitCode=0 Oct 30 21:26:14.857212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.857208 4988 generic.go:356] "Generic (PLEG): container finished" podID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerID="55a0dbaf2a8821f7489ee2389ae0aca365f7e6e5a715f9a1ded322d14ae23366" exitCode=0 Oct 30 21:26:14.857461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.857251 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerDied","Data":"208d0977763d09b121b2996752e392eb25af92863bae0c33b0aaf9222ca8e738"} Oct 30 21:26:14.857461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.857312 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerDied","Data":"55a0dbaf2a8821f7489ee2389ae0aca365f7e6e5a715f9a1ded322d14ae23366"} Oct 30 21:26:14.971180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:14.971150 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:26:15.057015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.056917 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.056973 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057046 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb\" (UniqueName: \"kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057075 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057120 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057455 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057324 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057455 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057368 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057455 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057425 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057448 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:15.057627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057471 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants\") pod \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\" (UID: \"d011eab9-c0d8-403e-85bc-96a5276c1fa9\") " Oct 30 21:26:15.057627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057554 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:15.057786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057751 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.057786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057773 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.057923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057813 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:15.057923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.057882 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:15.059560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.059535 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:15.059683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.059577 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "kube-api-access-fm2cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:15.059683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.059676 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:15.059923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.059906 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:15.060587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.060569 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "d011eab9-c0d8-403e-85bc-96a5276c1fa9" (UID: "d011eab9-c0d8-403e-85bc-96a5276c1fa9"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:26:15.158801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158756 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2cb\" (UniqueName: \"kubernetes.io/projected/d011eab9-c0d8-403e-85bc-96a5276c1fa9-kube-api-access-fm2cb\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.158801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158788 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.159078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158821 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:26:15.159078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158836 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.159078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158847 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.159078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158855 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.159078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.158863 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/d011eab9-c0d8-403e-85bc-96a5276c1fa9-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.168051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.168024 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:26:15.259641 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.259598 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 21:26:15.871898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.871859 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"d011eab9-c0d8-403e-85bc-96a5276c1fa9","Type":"ContainerDied","Data":"c116235466519e45158961420a10f454651c688243055dafb6ede811a30bd6cd"} Oct 30 21:26:15.871898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.871887 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:26:15.872432 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.871948 4988 scope.go:122] "RemoveContainer" containerID="b8205b11a6405bcb11c6e62feb8895f61c946a223db2dd6e2604d4be985ae477" Oct 30 21:26:15.880814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.880793 4988 scope.go:122] "RemoveContainer" containerID="208d0977763d09b121b2996752e392eb25af92863bae0c33b0aaf9222ca8e738" Oct 30 21:26:15.888876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:15.888853 4988 scope.go:122] "RemoveContainer" containerID="55a0dbaf2a8821f7489ee2389ae0aca365f7e6e5a715f9a1ded322d14ae23366" Oct 30 21:26:16.015487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:16.015450 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e768ca-7771-47ec-b47e-b395a7a4dcde" path="/var/lib/kubelet/pods/a7e768ca-7771-47ec-b47e-b395a7a4dcde/volumes" Oct 30 21:26:16.086895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:16.086856 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:26:16.137151 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:16.137069 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:26:18.014890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:18.014855 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" path="/var/lib/kubelet/pods/d011eab9-c0d8-403e-85bc-96a5276c1fa9/volumes" Oct 30 21:26:18.575731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:18.575697 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:26.394287 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:26:26.394206 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:26:26.541386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:26.541342 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:26.683510 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:26.683423 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:26.828758 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:26.828718 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:26.973392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:26.973297 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.116851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.116816 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.260869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.260783 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.411461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.411391 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.560538 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.560462 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.601055 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601018 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk"] Oct 30 21:26:27.601327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601316 4988 cpu_manager.go:414] "containerMap: removing container" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601328 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601339 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601343 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601351 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601356 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601366 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway-opa" Oct 30 21:26:27.601370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601370 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway-opa" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601388 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601394 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601403 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a7e768ca-7771-47ec-b47e-b395a7a4dcde" containerName="otc-container" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601420 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e768ca-7771-47ec-b47e-b395a7a4dcde" containerName="otc-container" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601469 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601478 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway-opa" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601486 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d011eab9-c0d8-403e-85bc-96a5276c1fa9" containerName="tempo-gateway" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601492 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e768ca-7771-47ec-b47e-b395a7a4dcde" containerName="otc-container" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601501 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd64e775-f161-4439-bf7b-5348c458676b" containerName="tempo" Oct 30 21:26:27.601599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.601506 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ecefc32-81b3-4831-a88c-a6bef976d270" containerName="tempo" Oct 30 21:26:27.606136 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.606118 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.659067 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.659031 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:26:27.712062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.712013 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-kts8k\"" Oct 30 21:26:27.712253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.712078 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:26:27.713295 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.713268 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk"] Oct 30 21:26:27.714288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.714268 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7"] Oct 30 21:26:27.717848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.717827 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.761870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.761834 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:26:27.762064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.761897 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:26:27.762064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.761942 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:26:27.762064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.761952 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:26:27.763219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.763195 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.763263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.763242 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.763306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.763261 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.763442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.763404 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.763479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.763453 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9g96\" (UniqueName: \"kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.772035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.772011 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-dspxv\"" Oct 30 21:26:27.773844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.773825 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:26:27.776252 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.776226 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7"] Oct 30 21:26:27.849900 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.849813 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:26:27.861337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.861305 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:27.864229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864205 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.864320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864239 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.864320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864275 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmdk\" (UniqueName: \"kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.864320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864312 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.864486 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864339 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.864486 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864360 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.864584 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864481 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9g96\" (UniqueName: \"kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.864584 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.864547 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.865054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.865025 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.865150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.865063 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.865150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.865131 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.866901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.866885 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.926189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.926156 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9g96\" (UniqueName: \"kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:27.965856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.965814 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.965856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.965864 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.966077 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.965913 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmdk\" (UniqueName: \"kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.966620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.966598 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:27.968341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:27.968320 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:28.033231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.033194 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmdk\" (UniqueName: \"kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk\") pod \"data-science-thanos-querier-proxy-77657866bd-df9l7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:28.034972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.034946 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:26:28.223225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.223173 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:26:28.411524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.411489 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7"] Oct 30 21:26:28.415030 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:26:28.414999 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83237bc5_a9b6_44af_b886_44efb2a8fbd7.slice/crio-0f4121cd2a0d57e5b107182ec17a99c341f585ef758ac075a75a73833490407a WatchSource:0}: Error finding container 0f4121cd2a0d57e5b107182ec17a99c341f585ef758ac075a75a73833490407a: Status 404 returned error can't find the container with id 0f4121cd2a0d57e5b107182ec17a99c341f585ef758ac075a75a73833490407a Oct 30 21:26:28.598957 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.598931 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk"] Oct 30 21:26:28.602089 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:26:28.602064 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d18b72_f340_4eb2_a148_429dfbc2c7a6.slice/crio-8dd7f3178727a15371819edc835d0b0adc92e3962f27d497e79b0c40edcc96ad WatchSource:0}: Error finding container 8dd7f3178727a15371819edc835d0b0adc92e3962f27d497e79b0c40edcc96ad: Status 404 returned error can't find the container with id 8dd7f3178727a15371819edc835d0b0adc92e3962f27d497e79b0c40edcc96ad Oct 30 21:26:28.935578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.935528 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerStarted","Data":"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f"} Oct 30 21:26:28.935796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.935651 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerStarted","Data":"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650"} Oct 30 21:26:28.935796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.935682 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerStarted","Data":"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336"} Oct 30 21:26:28.935796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.935705 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerStarted","Data":"8dd7f3178727a15371819edc835d0b0adc92e3962f27d497e79b0c40edcc96ad"} Oct 30 21:26:28.937522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.937496 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerStarted","Data":"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe"} Oct 30 21:26:28.937626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.937548 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerStarted","Data":"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc"} Oct 30 21:26:28.937626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.937572 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerStarted","Data":"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601"} Oct 30 21:26:28.937626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:28.937598 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerStarted","Data":"0f4121cd2a0d57e5b107182ec17a99c341f585ef758ac075a75a73833490407a"} Oct 30 21:26:29.054480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:29.054347 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" podStartSLOduration=2.054334166 podStartE2EDuration="2.054334166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:29.053224444 +0000 UTC m=+3017.160720368" watchObservedRunningTime="2026-10-30 21:26:29.054334166 +0000 UTC m=+3017.161830074" Oct 30 21:26:29.170985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:29.170930 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" podStartSLOduration=2.170914946 podStartE2EDuration="2.170914946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:29.169816052 +0000 UTC m=+3017.277311964" watchObservedRunningTime="2026-10-30 21:26:29.170914946 +0000 UTC m=+3017.278410854" Oct 30 21:26:29.222543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:29.222509 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:32.318469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:32.318430 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:32.584895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:32.584803 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:32.893540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:32.893506 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:44.386880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:44.386842 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:44.543023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:44.542982 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:44.716997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:44.716875 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:45.057167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:45.057045 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:45.204493 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:45.204454 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:45.354494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:45.354376 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:45.972088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:45.972048 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:46.127919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.127883 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:26:46.131852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.131828 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.178764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.178726 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:26:46.180622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.180594 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:26:46.200199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.200170 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:26:46.228495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.228356 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:26:46.228676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.228576 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:26:46.229989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.229961 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:26:46.233828 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.233803 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.288163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.288131 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:26:46.297087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297052 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:26:46.297087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297067 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297097 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297114 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297116 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297122 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297189 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dfb6w\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297202 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297324 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297331 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:26:46.297405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.297347 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-lg9b9\"" Oct 30 21:26:46.300588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.300572 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:26:46.414368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414325 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414374 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414395 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414511 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414534 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414594 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gm82\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414662 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414681 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414699 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414752 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414815 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.414925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.414832 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516236 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.516326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516289 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.516546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516349 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjw6\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.516546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516385 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516495 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.516646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516574 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516595 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516615 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516634 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516655 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.516827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516674 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gm82\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516693 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.516827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516717 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.516827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.516754 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.517158 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517133 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.517242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517190 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.517242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517229 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.517357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517254 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.517357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517285 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.517357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517308 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.517738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.517712 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.519751 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.519711 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.519875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.519779 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520040 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520572 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520553 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520655 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520568 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520655 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520632 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520650 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520705 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.520786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.520724 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.580939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.580902 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gm82\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:46.618707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618673 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.618914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618735 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.618914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618785 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.618914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618835 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.618914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618881 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.619131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618914 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.619131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.618944 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjw6\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.619336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.619285 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.621393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.621368 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.621680 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.621658 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.621745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.621660 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.621745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.621729 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.621853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.621754 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.680562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.680526 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjw6\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:46.850875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:46.850781 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:47.279464 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:47.279423 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:26:47.282588 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:26:47.282557 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ff9f2c_b555_4e4d_8d29_2020a5ef1ef8.slice/crio-517221bdae42899f30b402c12f2b6936eaeee949dbe52fded491192a7dd1ed8d WatchSource:0}: Error finding container 517221bdae42899f30b402c12f2b6936eaeee949dbe52fded491192a7dd1ed8d: Status 404 returned error can't find the container with id 517221bdae42899f30b402c12f2b6936eaeee949dbe52fded491192a7dd1ed8d Oct 30 21:26:48.034281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:48.034241 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerStarted","Data":"a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76"} Oct 30 21:26:48.034528 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:48.034308 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerStarted","Data":"517221bdae42899f30b402c12f2b6936eaeee949dbe52fded491192a7dd1ed8d"} Oct 30 21:26:51.049925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:51.049892 4988 generic.go:356] "Generic (PLEG): container finished" podID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerID="a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76" exitCode=0 Oct 30 21:26:51.050306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:51.049961 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerDied","Data":"a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76"} Oct 30 21:26:52.057053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:52.056964 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerStarted","Data":"66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1"} Oct 30 21:26:52.057053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:52.057034 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerStarted","Data":"6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea"} Oct 30 21:26:52.057511 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:52.057214 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:52.182848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:52.182790 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.896646765 podStartE2EDuration="6.1827743s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286127535s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:52.180818241 +0000 UTC m=+3040.288314142" watchObservedRunningTime="2026-10-30 21:26:52.1827743 +0000 UTC m=+3040.290270210" Oct 30 21:26:53.479326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.479275 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:53.584162 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.584128 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:53.681042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.680997 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:53.681236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.681128 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:53.733621 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.733534 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") DevicePath \"csi-468025a65b88095496ffee260c47731d5a0a7d1245692f39a473b974fd0f0891\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:53.875236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.875206 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fb02f91ab75afdcfd8c2ac3eaeb0ba7303c67490d38fe3164f5fc7856f2f1875/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:53.941908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:53.941874 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:54.253109 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:54.253070 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:26:54.739802 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:26:54.739760 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abef4a3_111c_4183_9aae_1836524b0fa1.slice/crio-9592783a5274486d9459c2c453faaa2464b608099e6623df94b2b75e6ea5f0c4 WatchSource:0}: Error finding container 9592783a5274486d9459c2c453faaa2464b608099e6623df94b2b75e6ea5f0c4: Status 404 returned error can't find the container with id 9592783a5274486d9459c2c453faaa2464b608099e6623df94b2b75e6ea5f0c4 Oct 30 21:26:54.786817 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:54.786781 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:26:55.071985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:55.071897 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerStarted","Data":"9592783a5274486d9459c2c453faaa2464b608099e6623df94b2b75e6ea5f0c4"} Oct 30 21:26:56.078068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:56.078028 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerStarted","Data":"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8"} Oct 30 21:26:56.395222 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:26:56.395185 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:26:58.063533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:58.063495 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:26:59.092524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:59.092489 4988 generic.go:356] "Generic (PLEG): container finished" podID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerID="60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8" exitCode=0 Oct 30 21:26:59.092903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:26:59.092562 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerDied","Data":"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8"} Oct 30 21:27:00.100737 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:00.100701 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerStarted","Data":"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c"} Oct 30 21:27:00.101143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:00.100773 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerStarted","Data":"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519"} Oct 30 21:27:00.101143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:00.100795 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerStarted","Data":"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8"} Oct 30 21:27:00.229115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:00.229058 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.443234678 podStartE2EDuration="14.229040633s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785805955s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:00.227970381 +0000 UTC m=+3048.335466289" watchObservedRunningTime="2026-10-30 21:27:00.229040633 +0000 UTC m=+3048.336536546" Oct 30 21:27:04.253911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:04.253868 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:27:09.253249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:09.253207 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:27:09.258749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:09.258724 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:27:10.150216 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:10.150184 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:27:18.920428 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:18.920378 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:23.152616 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:23.152579 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:24.788157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:24.788116 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:25.025611 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:25.025568 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:25.170911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:25.170876 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:25.318558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:25.318524 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:25.471355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:25.471270 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:25.619798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:25.619763 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:26.394870 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:27:26.394824 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:27:30.902047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:30.902005 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:31.102049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:31.102005 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:31.257185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:31.257097 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:33.438455 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.438398 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:27:33.443290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.443264 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.523639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.523596 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.523639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.523643 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.523914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.523700 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.523914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.523765 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.523914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.523855 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jn2\" (UniqueName: \"kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.524047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.523933 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.545792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.545759 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:27:33.545792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.545770 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:27:33.545792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.545776 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:27:33.546069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.545829 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-mpgx5\"" Oct 30 21:27:33.546069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.545846 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:27:33.546069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.545935 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:27:33.547597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.547567 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:27:33.624778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.624741 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jn2\" (UniqueName: \"kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.624991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.624799 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.624991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.624822 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.624991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.624844 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.624991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.624868 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.624991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.624904 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.625804 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.625774 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.627624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.627586 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.627769 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.627627 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.627769 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.627638 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.627769 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.627626 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.685949 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.685913 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jn2\" (UniqueName: \"kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2\") pod \"data-science-collector-collector-1\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:33.755993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:33.755900 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:27:34.139904 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:34.139876 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:27:34.142945 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:27:34.142912 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c533fca_307b_406f_b9f7_9e8b29bb2291.slice/crio-a06303f60bee093b6b0810160ef82dbc33945159c47b4c14f900c9de8c7fbbc8 WatchSource:0}: Error finding container a06303f60bee093b6b0810160ef82dbc33945159c47b4c14f900c9de8c7fbbc8: Status 404 returned error can't find the container with id a06303f60bee093b6b0810160ef82dbc33945159c47b4c14f900c9de8c7fbbc8 Oct 30 21:27:34.145307 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:34.145273 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:27:34.145363 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:34.145339 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:27:34.258038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:34.257998 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c533fca-307b-406f-b9f7-9e8b29bb2291","Type":"ContainerStarted","Data":"500b7078116adced7676bba3dfde74e5e2f8ce7a7521ec4a3998f3d57ab99d77"} Oct 30 21:27:34.258237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:34.258066 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c533fca-307b-406f-b9f7-9e8b29bb2291","Type":"ContainerStarted","Data":"a06303f60bee093b6b0810160ef82dbc33945159c47b4c14f900c9de8c7fbbc8"} Oct 30 21:27:34.377220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:34.377157 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.377139233 podStartE2EDuration="1.377139233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:34.375582125 +0000 UTC m=+3082.483078018" watchObservedRunningTime="2026-10-30 21:27:34.377139233 +0000 UTC m=+3082.484635143" Oct 30 21:27:53.547891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:53.547808 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:53.695778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:27:53.695734 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:56.394523 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:27:56.394484 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:28:01.700711 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:01.700671 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:07.595575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:07.595518 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:07.744761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:07.744724 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:19.220806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:19.220768 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:19.790490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:19.790456 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:19.938853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:19.938817 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:20.086228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:20.086134 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:20.250575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:20.250534 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:20.532765 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:20.532735 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:20.731312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:20.731278 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:26.222446 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:26.222397 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:28:26.222850 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:26.222607 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4c533fca-307b-406f-b9f7-9e8b29bb2291" containerName="otc-container" containerID="cri-o://500b7078116adced7676bba3dfde74e5e2f8ce7a7521ec4a3998f3d57ab99d77" gracePeriod=30 Oct 30 21:28:26.394653 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:28:26.394612 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:28:26.566570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:26.566481 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:26.765239 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:26.765194 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:26.927894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:26.927836 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:28.509140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.509095 4988 generic.go:356] "Generic (PLEG): container finished" podID="4c533fca-307b-406f-b9f7-9e8b29bb2291" containerID="500b7078116adced7676bba3dfde74e5e2f8ce7a7521ec4a3998f3d57ab99d77" exitCode=0 Oct 30 21:28:28.509658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.509226 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c533fca-307b-406f-b9f7-9e8b29bb2291","Type":"ContainerDied","Data":"500b7078116adced7676bba3dfde74e5e2f8ce7a7521ec4a3998f3d57ab99d77"} Oct 30 21:28:28.565281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.565244 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:28:28.600636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.600552 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs\") pod \"4c533fca-307b-406f-b9f7-9e8b29bb2291\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " Oct 30 21:28:28.600816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.600648 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs\") pod \"4c533fca-307b-406f-b9f7-9e8b29bb2291\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " Oct 30 21:28:28.600816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.600673 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2\" (UniqueName: \"kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2\") pod \"4c533fca-307b-406f-b9f7-9e8b29bb2291\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " Oct 30 21:28:28.600816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.600699 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal\") pod \"4c533fca-307b-406f-b9f7-9e8b29bb2291\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " Oct 30 21:28:28.600816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.600716 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs\") pod \"4c533fca-307b-406f-b9f7-9e8b29bb2291\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " Oct 30 21:28:28.600816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.600753 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert\") pod \"4c533fca-307b-406f-b9f7-9e8b29bb2291\" (UID: \"4c533fca-307b-406f-b9f7-9e8b29bb2291\") " Oct 30 21:28:28.601096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.601069 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal" pod "4c533fca-307b-406f-b9f7-9e8b29bb2291" (UID: "4c533fca-307b-406f-b9f7-9e8b29bb2291"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:28.602916 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.602884 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert" pod "4c533fca-307b-406f-b9f7-9e8b29bb2291" (UID: "4c533fca-307b-406f-b9f7-9e8b29bb2291"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:28.603044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.602942 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs" pod "4c533fca-307b-406f-b9f7-9e8b29bb2291" (UID: "4c533fca-307b-406f-b9f7-9e8b29bb2291"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:28.603044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.602975 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2" pod "4c533fca-307b-406f-b9f7-9e8b29bb2291" (UID: "4c533fca-307b-406f-b9f7-9e8b29bb2291"). InnerVolumeSpecName "kube-api-access-x4jn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:28.603163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.603146 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs" pod "4c533fca-307b-406f-b9f7-9e8b29bb2291" (UID: "4c533fca-307b-406f-b9f7-9e8b29bb2291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:28.603334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.603320 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs" pod "4c533fca-307b-406f-b9f7-9e8b29bb2291" (UID: "4c533fca-307b-406f-b9f7-9e8b29bb2291"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:28.702063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.702028 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:28.702063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.702062 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jn2\" (UniqueName: \"kubernetes.io/projected/4c533fca-307b-406f-b9f7-9e8b29bb2291-kube-api-access-x4jn2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:28.702278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.702078 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c533fca-307b-406f-b9f7-9e8b29bb2291-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:28.702278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.702091 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:28.702278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.702104 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:28.702278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.702116 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c533fca-307b-406f-b9f7-9e8b29bb2291-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:28.944500 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.944467 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:28:28.944873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.944861 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4c533fca-307b-406f-b9f7-9e8b29bb2291" containerName="otc-container" Oct 30 21:28:28.944924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.944876 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c533fca-307b-406f-b9f7-9e8b29bb2291" containerName="otc-container" Oct 30 21:28:28.944967 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.944963 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c533fca-307b-406f-b9f7-9e8b29bb2291" containerName="otc-container" Oct 30 21:28:28.948367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.948349 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:28.980387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:28.980345 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:29.000635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.000600 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 21:28:29.000814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.000682 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:28:29.000915 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.000896 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 21:28:29.000950 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.000933 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-dbhhl\"" Oct 30 21:28:29.011172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.011134 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:28:29.021620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.021589 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 21:28:29.050046 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.050004 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:28:29.050046 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.050016 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 21:28:29.050046 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.050038 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 21:28:29.105564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105531 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105565 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105594 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105655 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhpq\" (UniqueName: \"kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105690 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105804 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105841 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.105911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.105861 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.192009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.191969 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:29.207322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207242 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207284 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207316 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207339 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhpq\" (UniqueName: \"kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207356 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207396 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207451 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.207590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.207483 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.208163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.208132 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.208397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.208348 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.208397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.208388 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.208552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.208433 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.210054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.210022 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.210182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.210117 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.210182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.210131 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.268489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.268448 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhpq\" (UniqueName: \"kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:29.515318 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.515233 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:28:29.515318 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.515264 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c533fca-307b-406f-b9f7-9e8b29bb2291","Type":"ContainerDied","Data":"a06303f60bee093b6b0810160ef82dbc33945159c47b4c14f900c9de8c7fbbc8"} Oct 30 21:28:29.515899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.515332 4988 scope.go:122] "RemoveContainer" containerID="500b7078116adced7676bba3dfde74e5e2f8ce7a7521ec4a3998f3d57ab99d77" Oct 30 21:28:29.678949 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.678913 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:28:29.683116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:29.683089 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:28:30.019381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:30.019344 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c533fca-307b-406f-b9f7-9e8b29bb2291" path="/var/lib/kubelet/pods/4c533fca-307b-406f-b9f7-9e8b29bb2291/volumes" Oct 30 21:28:35.765264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:35.765220 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:35.869632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:35.869593 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:35.967054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:35.967011 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:35.967245 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:35.967167 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:36.019227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:36.019146 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:36.158271 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:36.158234 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:36.225949 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:36.225910 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:36.464404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:36.464366 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:37.025826 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:28:37.025787 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fdc678_e75e_46e2_86d1_161fe862691b.slice/crio-d12706e760eb8b7e6cce5bc9d2194e4859b3c76ff2c646ca6de258759dcc540b WatchSource:0}: Error finding container d12706e760eb8b7e6cce5bc9d2194e4859b3c76ff2c646ca6de258759dcc540b: Status 404 returned error can't find the container with id d12706e760eb8b7e6cce5bc9d2194e4859b3c76ff2c646ca6de258759dcc540b Oct 30 21:28:37.030179 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.030154 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:28:37.560709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.560670 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerStarted","Data":"c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631"} Oct 30 21:28:37.560911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.560736 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerStarted","Data":"d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746"} Oct 30 21:28:37.560911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.560761 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerStarted","Data":"477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca"} Oct 30 21:28:37.560911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.560784 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerStarted","Data":"d12706e760eb8b7e6cce5bc9d2194e4859b3c76ff2c646ca6de258759dcc540b"} Oct 30 21:28:37.560911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.560888 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:37.563520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.563494 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:37.686231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:37.686176 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=9.686163866 podStartE2EDuration="9.686163866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:37.683454289 +0000 UTC m=+3145.790950199" watchObservedRunningTime="2026-10-30 21:28:37.686163866 +0000 UTC m=+3145.793659777" Oct 30 21:28:38.358263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:38.358222 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:38.565192 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:38.565162 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:38.565774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:38.565212 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:38.572631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:38.572609 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:28:53.472571 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:53.472525 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:53.621185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:53.621147 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:53.771627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:53.771533 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:53.920267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:53.920226 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:54.068788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:54.068701 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:54.225045 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:54.225013 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:54.370700 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:54.370614 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:56.393629 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:28:56.393594 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:28:59.571864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:59.571826 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:59.715250 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:59.715213 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:28:59.923714 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:28:59.923665 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:00.579739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:00.579684 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:29:02.116541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.116504 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:02.143509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.143475 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:29:02.143754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.143701 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo" containerID="cri-o://477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca" gracePeriod=30 Oct 30 21:29:02.143822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.143787 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway" containerID="cri-o://d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746" gracePeriod=30 Oct 30 21:29:02.143913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.143759 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway-opa" containerID="cri-o://c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631" gracePeriod=30 Oct 30 21:29:02.146882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.146844 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:29:02.198219 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:29:02.198176 4988 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fdc678_e75e_46e2_86d1_161fe862691b.slice/crio-conmon-c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:29:02.561955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.561908 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.104:8083/ready\": dial tcp 10.132.0.104:8083: connect: connection refused" Oct 30 21:29:02.681869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.681828 4988 generic.go:356] "Generic (PLEG): container finished" podID="29fdc678-e75e-46e2-86d1-161fe862691b" containerID="c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631" exitCode=0 Oct 30 21:29:02.681869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.681855 4988 generic.go:356] "Generic (PLEG): container finished" podID="29fdc678-e75e-46e2-86d1-161fe862691b" containerID="477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca" exitCode=0 Oct 30 21:29:02.681869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.681878 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerDied","Data":"c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631"} Oct 30 21:29:02.682112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:02.681941 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerDied","Data":"477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca"} Oct 30 21:29:03.394116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.394087 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:03.508951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.508864 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.508951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.508911 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.508951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.508937 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq\" (UniqueName: \"kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509050 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509081 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509114 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509173 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509226 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509267 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf\") pod \"29fdc678-e75e-46e2-86d1-161fe862691b\" (UID: \"29fdc678-e75e-46e2-86d1-161fe862691b\") " Oct 30 21:29:03.509532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509317 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:03.509642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509595 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.509761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509716 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:03.509913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509861 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:03.509913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.509892 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:03.511733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.511694 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:03.511733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.511719 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:03.511733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.511703 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:03.511963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.511754 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "kube-api-access-tzhpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:03.512540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.512519 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "29fdc678-e75e-46e2-86d1-161fe862691b" (UID: "29fdc678-e75e-46e2-86d1-161fe862691b"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:29:03.610095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610044 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.610095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610084 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.610095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610100 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.610367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610113 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.610367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610125 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.610367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610137 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhpq\" (UniqueName: \"kubernetes.io/projected/29fdc678-e75e-46e2-86d1-161fe862691b-kube-api-access-tzhpq\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.610367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610174 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:29:03.610367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.610188 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/29fdc678-e75e-46e2-86d1-161fe862691b-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:03.619506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.619484 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:29:03.688243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.688208 4988 generic.go:356] "Generic (PLEG): container finished" podID="29fdc678-e75e-46e2-86d1-161fe862691b" containerID="d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746" exitCode=0 Oct 30 21:29:03.688435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.688254 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerDied","Data":"d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746"} Oct 30 21:29:03.688435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.688319 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"29fdc678-e75e-46e2-86d1-161fe862691b","Type":"ContainerDied","Data":"d12706e760eb8b7e6cce5bc9d2194e4859b3c76ff2c646ca6de258759dcc540b"} Oct 30 21:29:03.688435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.688269 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:03.688435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.688346 4988 scope.go:122] "RemoveContainer" containerID="c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631" Oct 30 21:29:03.698102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.698082 4988 scope.go:122] "RemoveContainer" containerID="d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746" Oct 30 21:29:03.706518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.706492 4988 scope.go:122] "RemoveContainer" containerID="477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca" Oct 30 21:29:03.711031 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.710998 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 21:29:03.714890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.714863 4988 scope.go:122] "RemoveContainer" containerID="c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631" Oct 30 21:29:03.715231 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:29:03.715206 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631\": container with ID starting with c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631 not found: ID does not exist" containerID="c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631" Oct 30 21:29:03.715300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.715244 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631"} err="failed to get container status \"c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631\": rpc error: code = NotFound desc = could not find container \"c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631\": container with ID starting with c00600a750f2fc8c55f4d5cf13e273c30dd6e1cfdcfbd888eb9d878bf8f4d631 not found: ID does not exist" Oct 30 21:29:03.715300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.715263 4988 scope.go:122] "RemoveContainer" containerID="d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746" Oct 30 21:29:03.715575 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:29:03.715549 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746\": container with ID starting with d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746 not found: ID does not exist" containerID="d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746" Oct 30 21:29:03.715625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.715582 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746"} err="failed to get container status \"d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746\": rpc error: code = NotFound desc = could not find container \"d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746\": container with ID starting with d76c38f7fe86d9748eaa994df8a8ed09012f35a1eaa86c8bdcaa251f42818746 not found: ID does not exist" Oct 30 21:29:03.715625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.715598 4988 scope.go:122] "RemoveContainer" containerID="477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca" Oct 30 21:29:03.715834 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:29:03.715812 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca\": container with ID starting with 477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca not found: ID does not exist" containerID="477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca" Oct 30 21:29:03.715880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.715839 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca"} err="failed to get container status \"477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca\": rpc error: code = NotFound desc = could not find container \"477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca\": container with ID starting with 477fe842ded6f55ebe0f15ecf6a3d562cb25dc82e71fc46d6e15ac08ea6b8aca not found: ID does not exist" Oct 30 21:29:03.853148 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.853057 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:29:03.857809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:03.857771 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:29:04.018598 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:04.018568 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" path="/var/lib/kubelet/pods/29fdc678-e75e-46e2-86d1-161fe862691b/volumes" Oct 30 21:29:19.564090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:19.564058 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:26.395458 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:29:26.395351 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:29:26.434749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:26.434708 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:26.579809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:26.579755 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:26.793021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:26.792902 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:26.943164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:26.943126 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:27.087817 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:27.087734 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:27.229206 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:27.229165 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:32.866108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:32.866069 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:33.067642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:33.067605 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:33.223029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:33.222987 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:34.984082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984044 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:29:34.984527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984441 4988 cpu_manager.go:414] "containerMap: removing container" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo" Oct 30 21:29:34.984527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984452 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo" Oct 30 21:29:34.984527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984461 4988 cpu_manager.go:414] "containerMap: removing container" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway-opa" Oct 30 21:29:34.984527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984483 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway-opa" Oct 30 21:29:34.984527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984492 4988 cpu_manager.go:414] "containerMap: removing container" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway" Oct 30 21:29:34.984527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984497 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway" Oct 30 21:29:34.984747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984553 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway" Oct 30 21:29:34.984747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984563 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo" Oct 30 21:29:34.984747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.984568 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fdc678-e75e-46e2-86d1-161fe862691b" containerName="tempo-gateway-opa" Oct 30 21:29:34.989388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.989353 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:34.997144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:34.997107 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:35.041802 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.041769 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:29:35.041802 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.041769 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 21:29:35.042029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.041851 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 21:29:35.091329 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.091292 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-t67zt\"" Oct 30 21:29:35.091585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.091354 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 21:29:35.091585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.091356 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 21:29:35.091585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.091356 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 21:29:35.091585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.091467 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 21:29:35.093299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.093273 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:29:35.199732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199691 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.199732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199738 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.199948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199816 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.199948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199860 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.199948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199916 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.199948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199939 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4st\" (UniqueName: \"kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.200082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.199967 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.200082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.200017 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.289575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.289468 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:35.300612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.300578 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.300728 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.300618 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.300864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.300837 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.300928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.300908 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.300992 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.300940 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.300992 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.300964 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.301095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.301007 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.301265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.301241 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4st\" (UniqueName: \"kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.301478 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.301402 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.301602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.301501 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.301602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.301534 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.301723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.301680 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.303566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.303535 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.303672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.303537 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.303672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.303599 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:35.362695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:35.362653 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4st\" (UniqueName: \"kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.423791 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.423738 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.527696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.527656 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.626170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.626112 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.626400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.626274 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.684080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.683972 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.823761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.823727 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9da1b29e6f97cbef8aa0879e8d7bcaa22b72d8e13243848f419dad2d056ca757/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:37.891337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:37.891303 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:38.009108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.009004 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:38.491861 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:29:38.491827 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c67e49e_c23e_4dd0_af65_e8232ff06103.slice/crio-9550205fbe6c63c19e5db3c7467efc49ec19a1f27f864ddd54d34ab3eb7ad8f4 WatchSource:0}: Error finding container 9550205fbe6c63c19e5db3c7467efc49ec19a1f27f864ddd54d34ab3eb7ad8f4: Status 404 returned error can't find the container with id 9550205fbe6c63c19e5db3c7467efc49ec19a1f27f864ddd54d34ab3eb7ad8f4 Oct 30 21:29:38.538092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.538059 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:29:38.850523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.850443 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerStarted","Data":"3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287"} Oct 30 21:29:38.850676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.850536 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerStarted","Data":"0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc"} Oct 30 21:29:38.850676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.850564 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerStarted","Data":"1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40"} Oct 30 21:29:38.850676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.850587 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerStarted","Data":"9550205fbe6c63c19e5db3c7467efc49ec19a1f27f864ddd54d34ab3eb7ad8f4"} Oct 30 21:29:38.850676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.850624 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:38.850676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.850648 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:38.853525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.853502 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:38.859579 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.859537 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:38.975170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:38.975116 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.975101455 podStartE2EDuration="4.975101455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:38.973039104 +0000 UTC m=+3207.080535014" watchObservedRunningTime="2026-10-30 21:29:38.975101455 +0000 UTC m=+3207.082597365" Oct 30 21:29:39.854751 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:39.854719 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:29:50.545881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:29:50.545843 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:29:56.394623 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:29:56.394579 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:30:00.861925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:00.861875 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:30:00.932057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:00.932019 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:01.128058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:01.127967 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:06.007148 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:06.007102 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:06.214545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:06.214506 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:06.379009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:06.378922 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:08.558202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.558162 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:30:08.558666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.558473 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo" containerID="cri-o://1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40" gracePeriod=30 Oct 30 21:30:08.558666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.558492 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway-opa" containerID="cri-o://3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287" gracePeriod=30 Oct 30 21:30:08.558666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.558502 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway" containerID="cri-o://0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc" gracePeriod=30 Oct 30 21:30:08.560269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.560213 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:30:08.579129 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.579095 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:08.851074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.850974 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.132.0.105:8083/ready\": dial tcp 10.132.0.105:8083: connect: connection refused" Oct 30 21:30:08.854968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.854933 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:30:08.989686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.989652 4988 generic.go:356] "Generic (PLEG): container finished" podID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerID="3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287" exitCode=0 Oct 30 21:30:08.989686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.989679 4988 generic.go:356] "Generic (PLEG): container finished" podID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerID="1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40" exitCode=0 Oct 30 21:30:08.989911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.989727 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerDied","Data":"3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287"} Oct 30 21:30:08.989911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:08.989779 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerDied","Data":"1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40"} Oct 30 21:30:09.809933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.809902 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:30:09.811996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.811961 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812039 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812073 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812100 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812252 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812198 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812252 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812225 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812274 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st\" (UniqueName: \"kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812307 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812345 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf\") pod \"9c67e49e-c23e-4dd0-af65-e8232ff06103\" (UID: \"9c67e49e-c23e-4dd0-af65-e8232ff06103\") " Oct 30 21:30:09.812599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812556 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:09.812679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812658 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:09.812994 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812935 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:09.812994 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.812947 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:09.814400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.814378 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:09.814762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.814736 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "kube-api-access-vz4st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:09.814868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.814763 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:09.814868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.814840 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:09.815115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.815098 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06" pod "9c67e49e-c23e-4dd0-af65-e8232ff06103" (UID: "9c67e49e-c23e-4dd0-af65-e8232ff06103"). InnerVolumeSpecName "pvc-e7fabdfe-f269-407c-827e-7980f4596959". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913192 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4st\" (UniqueName: \"kubernetes.io/projected/9c67e49e-c23e-4dd0-af65-e8232ff06103-kube-api-access-vz4st\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913222 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913233 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-conf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913243 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913253 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913264 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913275 4988 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-gateway-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.913659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913303 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:30:09.913659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.913312 4988 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c67e49e-c23e-4dd0-af65-e8232ff06103-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:09.922366 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.922346 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e7fabdfe-f269-407c-827e-7980f4596959" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:30:09.997370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.997330 4988 generic.go:356] "Generic (PLEG): container finished" podID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerID="0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc" exitCode=0 Oct 30 21:30:09.997570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.997394 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 21:30:09.997570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.997508 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerDied","Data":"0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc"} Oct 30 21:30:09.997646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.997586 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c67e49e-c23e-4dd0-af65-e8232ff06103","Type":"ContainerDied","Data":"9550205fbe6c63c19e5db3c7467efc49ec19a1f27f864ddd54d34ab3eb7ad8f4"} Oct 30 21:30:09.997646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:09.997611 4988 scope.go:122] "RemoveContainer" containerID="3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287" Oct 30 21:30:10.006612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.006585 4988 scope.go:122] "RemoveContainer" containerID="0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc" Oct 30 21:30:10.013725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.013696 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e7fabdfe-f269-407c-827e-7980f4596959\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0de7fcc7e4205ee06\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-f69c97d9d5b15d49f77f18dd0d865f0012945066238ead41e186d325f4912f5b\"" Oct 30 21:30:10.015918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.015901 4988 scope.go:122] "RemoveContainer" containerID="1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40" Oct 30 21:30:10.024636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.024609 4988 scope.go:122] "RemoveContainer" containerID="3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287" Oct 30 21:30:10.024984 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:10.024963 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287\": container with ID starting with 3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287 not found: ID does not exist" containerID="3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287" Oct 30 21:30:10.025080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.024996 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287"} err="failed to get container status \"3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287\": rpc error: code = NotFound desc = could not find container \"3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287\": container with ID starting with 3118818f688ae0b65a63969790825fcb613f56db6369a9a69b612514c3a33287 not found: ID does not exist" Oct 30 21:30:10.025080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.025020 4988 scope.go:122] "RemoveContainer" containerID="0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc" Oct 30 21:30:10.025313 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:10.025292 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc\": container with ID starting with 0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc not found: ID does not exist" containerID="0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc" Oct 30 21:30:10.025353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.025319 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc"} err="failed to get container status \"0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc\": rpc error: code = NotFound desc = could not find container \"0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc\": container with ID starting with 0e935fae06e10e2bf33735bdcfa94eee9e948e13a26bef2ca85e51bbae92b1cc not found: ID does not exist" Oct 30 21:30:10.025353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.025334 4988 scope.go:122] "RemoveContainer" containerID="1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40" Oct 30 21:30:10.025605 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:10.025588 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40\": container with ID starting with 1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40 not found: ID does not exist" containerID="1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40" Oct 30 21:30:10.025665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.025609 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40"} err="failed to get container status \"1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40\": rpc error: code = NotFound desc = could not find container \"1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40\": container with ID starting with 1476abf63c214fe50063dcadaf235f4c78762e99570fbc84ec1636d159eb7f40 not found: ID does not exist" Oct 30 21:30:10.171171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.171081 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:30:10.177449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:10.177405 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 21:30:12.017283 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:12.017252 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" path="/var/lib/kubelet/pods/9c67e49e-c23e-4dd0-af65-e8232ff06103/volumes" Oct 30 21:30:19.914957 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:19.914920 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:24.222530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.222489 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk"] Oct 30 21:30:24.222910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.222755 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="kube-rbac-proxy" containerID="cri-o://6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" gracePeriod=30 Oct 30 21:30:24.222910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.222825 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="method-gate" containerID="cri-o://247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" gracePeriod=30 Oct 30 21:30:24.223052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.222829 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="prom-label-proxy" containerID="cri-o://1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" gracePeriod=30 Oct 30 21:30:24.279277 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.279242 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7"] Oct 30 21:30:24.279558 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.279527 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="prom-label-proxy" containerID="cri-o://efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" gracePeriod=30 Oct 30 21:30:24.279651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.279517 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="kube-rbac-proxy" containerID="cri-o://3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" gracePeriod=30 Oct 30 21:30:24.279717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.279537 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="method-gate" containerID="cri-o://7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" gracePeriod=30 Oct 30 21:30:24.530420 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.530382 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:30:24.558530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.558497 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:30:24.647970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.647936 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config\") pod \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " Oct 30 21:30:24.647970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.647969 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls\" (UniqueName: \"kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls\") pod \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " Oct 30 21:30:24.648229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.647998 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca\") pod \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " Oct 30 21:30:24.648229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648123 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca\") pod \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " Oct 30 21:30:24.648229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648168 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk\" (UniqueName: \"kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk\") pod \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " Oct 30 21:30:24.648229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648191 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config\" (UniqueName: \"kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config\") pod \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " Oct 30 21:30:24.648229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648221 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96\" (UniqueName: \"kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96\") pod \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\" (UID: \"11d18b72-f340-4eb2-a148-429dfbc2c7a6\") " Oct 30 21:30:24.648545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648286 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls\" (UniqueName: \"kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls\") pod \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\" (UID: \"83237bc5-a9b6-44af-b886-44efb2a8fbd7\") " Oct 30 21:30:24.648545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648307 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca" pod "11d18b72-f340-4eb2-a148-429dfbc2c7a6" (UID: "11d18b72-f340-4eb2-a148-429dfbc2c7a6"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.648545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648320 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config" pod "11d18b72-f340-4eb2-a148-429dfbc2c7a6" (UID: "11d18b72-f340-4eb2-a148-429dfbc2c7a6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.648746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648562 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.648746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648584 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.648746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648707 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config" pod "83237bc5-a9b6-44af-b886-44efb2a8fbd7" (UID: "83237bc5-a9b6-44af-b886-44efb2a8fbd7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.648981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.648844 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca" pod "11d18b72-f340-4eb2-a148-429dfbc2c7a6" (UID: "11d18b72-f340-4eb2-a148-429dfbc2c7a6"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.651133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.651097 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls" pod "83237bc5-a9b6-44af-b886-44efb2a8fbd7" (UID: "83237bc5-a9b6-44af-b886-44efb2a8fbd7"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:24.651257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.651176 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls" pod "11d18b72-f340-4eb2-a148-429dfbc2c7a6" (UID: "11d18b72-f340-4eb2-a148-429dfbc2c7a6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:24.651664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.651639 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk" pod "83237bc5-a9b6-44af-b886-44efb2a8fbd7" (UID: "83237bc5-a9b6-44af-b886-44efb2a8fbd7"). InnerVolumeSpecName "kube-api-access-5kmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.651931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.651902 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96" pod "11d18b72-f340-4eb2-a148-429dfbc2c7a6" (UID: "11d18b72-f340-4eb2-a148-429dfbc2c7a6"). InnerVolumeSpecName "kube-api-access-d9g96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.749513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.749398 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/83237bc5-a9b6-44af-b886-44efb2a8fbd7-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.749513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.749452 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/11d18b72-f340-4eb2-a148-429dfbc2c7a6-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.749513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.749465 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/11d18b72-f340-4eb2-a148-429dfbc2c7a6-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.749513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.749478 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmdk\" (UniqueName: \"kubernetes.io/projected/83237bc5-a9b6-44af-b886-44efb2a8fbd7-kube-api-access-5kmdk\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.749513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.749492 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83237bc5-a9b6-44af-b886-44efb2a8fbd7-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.749513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:24.749504 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9g96\" (UniqueName: \"kubernetes.io/projected/11d18b72-f340-4eb2-a148-429dfbc2c7a6-kube-api-access-d9g96\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:25.068805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068723 4988 generic.go:356] "Generic (PLEG): container finished" podID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerID="1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" exitCode=0 Oct 30 21:30:25.068805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068747 4988 generic.go:356] "Generic (PLEG): container finished" podID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerID="247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" exitCode=0 Oct 30 21:30:25.068805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068755 4988 generic.go:356] "Generic (PLEG): container finished" podID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerID="6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" exitCode=0 Oct 30 21:30:25.068805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068777 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" Oct 30 21:30:25.069106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068810 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerDied","Data":"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f"} Oct 30 21:30:25.069106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068885 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerDied","Data":"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650"} Oct 30 21:30:25.069106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068920 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerDied","Data":"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336"} Oct 30 21:30:25.069106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068938 4988 scope.go:122] "RemoveContainer" containerID="1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" Oct 30 21:30:25.069106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.068956 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk" event={"ID":"11d18b72-f340-4eb2-a148-429dfbc2c7a6","Type":"ContainerDied","Data":"8dd7f3178727a15371819edc835d0b0adc92e3962f27d497e79b0c40edcc96ad"} Oct 30 21:30:25.070849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.070825 4988 generic.go:356] "Generic (PLEG): container finished" podID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerID="efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" exitCode=0 Oct 30 21:30:25.070849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.070848 4988 generic.go:356] "Generic (PLEG): container finished" podID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerID="7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" exitCode=0 Oct 30 21:30:25.071010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.070854 4988 generic.go:356] "Generic (PLEG): container finished" podID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerID="3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" exitCode=0 Oct 30 21:30:25.071010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.070901 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" Oct 30 21:30:25.071010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.070917 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerDied","Data":"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe"} Oct 30 21:30:25.071010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.070981 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerDied","Data":"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc"} Oct 30 21:30:25.071010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.071006 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerDied","Data":"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601"} Oct 30 21:30:25.071169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.071028 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7" event={"ID":"83237bc5-a9b6-44af-b886-44efb2a8fbd7","Type":"ContainerDied","Data":"0f4121cd2a0d57e5b107182ec17a99c341f585ef758ac075a75a73833490407a"} Oct 30 21:30:25.080030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.080005 4988 scope.go:122] "RemoveContainer" containerID="247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" Oct 30 21:30:25.087925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.087904 4988 scope.go:122] "RemoveContainer" containerID="6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" Oct 30 21:30:25.095972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.095951 4988 scope.go:122] "RemoveContainer" containerID="1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" Oct 30 21:30:25.096317 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:25.096296 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": container with ID starting with 1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f not found: ID does not exist" containerID="1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" Oct 30 21:30:25.096370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.096329 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f"} err="failed to get container status \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": rpc error: code = NotFound desc = could not find container \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": container with ID starting with 1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f not found: ID does not exist" Oct 30 21:30:25.096370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.096347 4988 scope.go:122] "RemoveContainer" containerID="247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" Oct 30 21:30:25.096657 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:25.096628 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": container with ID starting with 247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650 not found: ID does not exist" containerID="247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" Oct 30 21:30:25.096743 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.096665 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650"} err="failed to get container status \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": rpc error: code = NotFound desc = could not find container \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": container with ID starting with 247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650 not found: ID does not exist" Oct 30 21:30:25.096743 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.096685 4988 scope.go:122] "RemoveContainer" containerID="6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" Oct 30 21:30:25.096913 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:25.096897 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": container with ID starting with 6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336 not found: ID does not exist" containerID="6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" Oct 30 21:30:25.096952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.096920 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336"} err="failed to get container status \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": rpc error: code = NotFound desc = could not find container \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": container with ID starting with 6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336 not found: ID does not exist" Oct 30 21:30:25.096952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.096933 4988 scope.go:122] "RemoveContainer" containerID="1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" Oct 30 21:30:25.097180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097160 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f"} err="failed to get container status \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": rpc error: code = NotFound desc = could not find container \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": container with ID starting with 1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f not found: ID does not exist" Oct 30 21:30:25.097244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097181 4988 scope.go:122] "RemoveContainer" containerID="247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" Oct 30 21:30:25.097370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097350 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650"} err="failed to get container status \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": rpc error: code = NotFound desc = could not find container \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": container with ID starting with 247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650 not found: ID does not exist" Oct 30 21:30:25.097370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097367 4988 scope.go:122] "RemoveContainer" containerID="6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" Oct 30 21:30:25.097605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097583 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336"} err="failed to get container status \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": rpc error: code = NotFound desc = could not find container \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": container with ID starting with 6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336 not found: ID does not exist" Oct 30 21:30:25.097652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097606 4988 scope.go:122] "RemoveContainer" containerID="1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f" Oct 30 21:30:25.097809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097789 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f"} err="failed to get container status \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": rpc error: code = NotFound desc = could not find container \"1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f\": container with ID starting with 1b9a7e0590420d6a76c8ad6248ed11ed700a130babf756038b3ca990fe33aa8f not found: ID does not exist" Oct 30 21:30:25.097873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.097810 4988 scope.go:122] "RemoveContainer" containerID="247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650" Oct 30 21:30:25.098034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.098016 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650"} err="failed to get container status \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": rpc error: code = NotFound desc = could not find container \"247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650\": container with ID starting with 247f4b61267b863a6d60c43287794d36e1b583c8fe57c20235f1dd2750039650 not found: ID does not exist" Oct 30 21:30:25.098079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.098035 4988 scope.go:122] "RemoveContainer" containerID="6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336" Oct 30 21:30:25.098243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.098225 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336"} err="failed to get container status \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": rpc error: code = NotFound desc = could not find container \"6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336\": container with ID starting with 6b228bc6ce475344300efe45388840ba54cf62850e842ae0ed981e5907c0f336 not found: ID does not exist" Oct 30 21:30:25.098315 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.098243 4988 scope.go:122] "RemoveContainer" containerID="efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" Oct 30 21:30:25.106237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.106177 4988 scope.go:122] "RemoveContainer" containerID="7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" Oct 30 21:30:25.114395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.114374 4988 scope.go:122] "RemoveContainer" containerID="3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" Oct 30 21:30:25.122348 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.122321 4988 scope.go:122] "RemoveContainer" containerID="efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" Oct 30 21:30:25.122691 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:25.122669 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": container with ID starting with efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe not found: ID does not exist" containerID="efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" Oct 30 21:30:25.122792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.122703 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe"} err="failed to get container status \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": rpc error: code = NotFound desc = could not find container \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": container with ID starting with efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe not found: ID does not exist" Oct 30 21:30:25.122792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.122726 4988 scope.go:122] "RemoveContainer" containerID="7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" Oct 30 21:30:25.122982 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:25.122957 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": container with ID starting with 7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc not found: ID does not exist" containerID="7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" Oct 30 21:30:25.123021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.122989 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc"} err="failed to get container status \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": rpc error: code = NotFound desc = could not find container \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": container with ID starting with 7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc not found: ID does not exist" Oct 30 21:30:25.123021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123003 4988 scope.go:122] "RemoveContainer" containerID="3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" Oct 30 21:30:25.123231 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:25.123214 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": container with ID starting with 3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601 not found: ID does not exist" containerID="3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" Oct 30 21:30:25.123292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123234 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601"} err="failed to get container status \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": rpc error: code = NotFound desc = could not find container \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": container with ID starting with 3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601 not found: ID does not exist" Oct 30 21:30:25.123292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123248 4988 scope.go:122] "RemoveContainer" containerID="efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" Oct 30 21:30:25.123501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123426 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe"} err="failed to get container status \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": rpc error: code = NotFound desc = could not find container \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": container with ID starting with efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe not found: ID does not exist" Oct 30 21:30:25.123501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123450 4988 scope.go:122] "RemoveContainer" containerID="7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" Oct 30 21:30:25.123686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123668 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc"} err="failed to get container status \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": rpc error: code = NotFound desc = could not find container \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": container with ID starting with 7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc not found: ID does not exist" Oct 30 21:30:25.123732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123686 4988 scope.go:122] "RemoveContainer" containerID="3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" Oct 30 21:30:25.123903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123883 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601"} err="failed to get container status \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": rpc error: code = NotFound desc = could not find container \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": container with ID starting with 3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601 not found: ID does not exist" Oct 30 21:30:25.123966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.123912 4988 scope.go:122] "RemoveContainer" containerID="efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe" Oct 30 21:30:25.124184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.124165 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe"} err="failed to get container status \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": rpc error: code = NotFound desc = could not find container \"efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe\": container with ID starting with efedaa1b4cca8381b310de0c3c45fa08c6d0d8dd693503d0091465905bb0c8fe not found: ID does not exist" Oct 30 21:30:25.124231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.124185 4988 scope.go:122] "RemoveContainer" containerID="7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc" Oct 30 21:30:25.124457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.124441 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc"} err="failed to get container status \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": rpc error: code = NotFound desc = could not find container \"7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc\": container with ID starting with 7e5188296d6776abb13d63793c3848806d8b5768feefbb6e8d837b1c332b12dc not found: ID does not exist" Oct 30 21:30:25.124457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.124456 4988 scope.go:122] "RemoveContainer" containerID="3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601" Oct 30 21:30:25.124686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.124667 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601"} err="failed to get container status \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": rpc error: code = NotFound desc = could not find container \"3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601\": container with ID starting with 3346f3b8bf2302c27af1a534dccdbc6c4d21c5593467e5d0d0dd8ac1c1e92601 not found: ID does not exist" Oct 30 21:30:25.247292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.247255 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7"] Oct 30 21:30:25.252907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.252879 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-df9l7"] Oct 30 21:30:25.355066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.354930 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:25.390388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.390349 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk"] Oct 30 21:30:25.394818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.394784 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-8s2nk"] Oct 30 21:30:25.497973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.497935 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:25.583093 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.583058 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:30:25.583346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.583317 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="alertmanager" containerID="cri-o://6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea" gracePeriod=120 Oct 30 21:30:25.583465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.583321 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="config-reloader" containerID="cri-o://66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1" gracePeriod=120 Oct 30 21:30:25.586097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.586071 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:30:25.586393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.586369 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="prometheus" containerID="cri-o://e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" gracePeriod=600 Oct 30 21:30:25.586544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.586509 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="config-reloader" containerID="cri-o://d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" gracePeriod=600 Oct 30 21:30:25.586678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.586519 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="thanos-sidecar" containerID="cri-o://e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" gracePeriod=600 Oct 30 21:30:25.652530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.652497 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:25.822379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.822341 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:25.844068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.844042 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:30:25.959873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.959832 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.959873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.959878 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960151 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.959906 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960151 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960083 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960150 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960187 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960233 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960266 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960273 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:25.960564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960542 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960590 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960623 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960680 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.960725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960694 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:25.960837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.960778 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") pod \"7abef4a3-111c-4183-9aae-1836524b0fa1\" (UID: \"7abef4a3-111c-4183-9aae-1836524b0fa1\") " Oct 30 21:30:25.961157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.961081 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:25.961157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.961108 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:25.961157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.961111 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:25.962792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.962771 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:25.962975 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.962954 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:25.963066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.962985 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "kube-api-access-7gm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:25.963066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.962996 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:25.963183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.963168 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:25.963635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.963609 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:25.964248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.964223 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:25.964328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.964315 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:25.967454 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.967437 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "pvc-e73afc79-aa37-49a0-8744-633c03cad1f4". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:30:25.976205 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.976170 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config" pod "7abef4a3-111c-4183-9aae-1836524b0fa1" (UID: "7abef4a3-111c-4183-9aae-1836524b0fa1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:25.977622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:25.977591 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:26.015353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.015311 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" path="/var/lib/kubelet/pods/11d18b72-f340-4eb2-a148-429dfbc2c7a6/volumes" Oct 30 21:30:26.016041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.016019 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" path="/var/lib/kubelet/pods/83237bc5-a9b6-44af-b886-44efb2a8fbd7/volumes" Oct 30 21:30:26.062076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062035 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062078 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gm82\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-kube-api-access-7gm82\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062094 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062111 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062123 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062136 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7abef4a3-111c-4183-9aae-1836524b0fa1-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062152 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062167 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7abef4a3-111c-4183-9aae-1836524b0fa1-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062183 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7abef4a3-111c-4183-9aae-1836524b0fa1-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062198 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7abef4a3-111c-4183-9aae-1836524b0fa1-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.062350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.062234 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:30:26.073603 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.073576 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e73afc79-aa37-49a0-8744-633c03cad1f4" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:30:26.078522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078489 4988 generic.go:356] "Generic (PLEG): container finished" podID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerID="e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" exitCode=0 Oct 30 21:30:26.078522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078514 4988 generic.go:356] "Generic (PLEG): container finished" podID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerID="d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" exitCode=0 Oct 30 21:30:26.078522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078520 4988 generic.go:356] "Generic (PLEG): container finished" podID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerID="e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" exitCode=0 Oct 30 21:30:26.078742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078566 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 21:30:26.078742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078565 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerDied","Data":"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c"} Oct 30 21:30:26.078742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078710 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerDied","Data":"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519"} Oct 30 21:30:26.078742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078742 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerDied","Data":"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8"} Oct 30 21:30:26.078881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078764 4988 scope.go:122] "RemoveContainer" containerID="e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" Oct 30 21:30:26.078881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.078776 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7abef4a3-111c-4183-9aae-1836524b0fa1","Type":"ContainerDied","Data":"9592783a5274486d9459c2c453faaa2464b608099e6623df94b2b75e6ea5f0c4"} Oct 30 21:30:26.081040 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.081013 4988 generic.go:356] "Generic (PLEG): container finished" podID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerID="66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1" exitCode=0 Oct 30 21:30:26.081143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.081115 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerDied","Data":"66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1"} Oct 30 21:30:26.087870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.087845 4988 scope.go:122] "RemoveContainer" containerID="d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" Oct 30 21:30:26.096716 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.096691 4988 scope.go:122] "RemoveContainer" containerID="e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" Oct 30 21:30:26.105304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.105279 4988 scope.go:122] "RemoveContainer" containerID="60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8" Oct 30 21:30:26.114060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114038 4988 scope.go:122] "RemoveContainer" containerID="e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" Oct 30 21:30:26.114389 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:26.114364 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": container with ID starting with e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c not found: ID does not exist" containerID="e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" Oct 30 21:30:26.114479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114402 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c"} err="failed to get container status \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": rpc error: code = NotFound desc = could not find container \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": container with ID starting with e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c not found: ID does not exist" Oct 30 21:30:26.114479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114437 4988 scope.go:122] "RemoveContainer" containerID="d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" Oct 30 21:30:26.114697 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:26.114680 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": container with ID starting with d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519 not found: ID does not exist" containerID="d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" Oct 30 21:30:26.114742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114706 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519"} err="failed to get container status \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": rpc error: code = NotFound desc = could not find container \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": container with ID starting with d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519 not found: ID does not exist" Oct 30 21:30:26.114742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114721 4988 scope.go:122] "RemoveContainer" containerID="e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" Oct 30 21:30:26.114937 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:26.114922 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": container with ID starting with e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8 not found: ID does not exist" containerID="e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" Oct 30 21:30:26.114976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114940 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8"} err="failed to get container status \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": rpc error: code = NotFound desc = could not find container \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": container with ID starting with e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8 not found: ID does not exist" Oct 30 21:30:26.114976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.114952 4988 scope.go:122] "RemoveContainer" containerID="60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8" Oct 30 21:30:26.115145 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:26.115129 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": container with ID starting with 60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8 not found: ID does not exist" containerID="60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8" Oct 30 21:30:26.115184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115148 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8"} err="failed to get container status \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": rpc error: code = NotFound desc = could not find container \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": container with ID starting with 60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8 not found: ID does not exist" Oct 30 21:30:26.115184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115160 4988 scope.go:122] "RemoveContainer" containerID="e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" Oct 30 21:30:26.115354 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115338 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c"} err="failed to get container status \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": rpc error: code = NotFound desc = could not find container \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": container with ID starting with e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c not found: ID does not exist" Oct 30 21:30:26.115392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115354 4988 scope.go:122] "RemoveContainer" containerID="d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" Oct 30 21:30:26.115661 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115645 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519"} err="failed to get container status \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": rpc error: code = NotFound desc = could not find container \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": container with ID starting with d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519 not found: ID does not exist" Oct 30 21:30:26.115706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115662 4988 scope.go:122] "RemoveContainer" containerID="e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" Oct 30 21:30:26.115842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115826 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8"} err="failed to get container status \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": rpc error: code = NotFound desc = could not find container \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": container with ID starting with e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8 not found: ID does not exist" Oct 30 21:30:26.115884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115842 4988 scope.go:122] "RemoveContainer" containerID="60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8" Oct 30 21:30:26.116013 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.115998 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8"} err="failed to get container status \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": rpc error: code = NotFound desc = could not find container \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": container with ID starting with 60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8 not found: ID does not exist" Oct 30 21:30:26.116013 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116012 4988 scope.go:122] "RemoveContainer" containerID="e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c" Oct 30 21:30:26.116179 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116160 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c"} err="failed to get container status \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": rpc error: code = NotFound desc = could not find container \"e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c\": container with ID starting with e8d66c22f9a068b8b387c8cdab6569bc425bbc79fec66f5cf06e8a29e5217f5c not found: ID does not exist" Oct 30 21:30:26.116221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116179 4988 scope.go:122] "RemoveContainer" containerID="d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519" Oct 30 21:30:26.116337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116317 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519"} err="failed to get container status \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": rpc error: code = NotFound desc = could not find container \"d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519\": container with ID starting with d41323e73844d6a179b847b62a5f3d3ecaa36f881cf61e90e1c9b8c3c556d519 not found: ID does not exist" Oct 30 21:30:26.116388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116337 4988 scope.go:122] "RemoveContainer" containerID="e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8" Oct 30 21:30:26.116532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116516 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8"} err="failed to get container status \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": rpc error: code = NotFound desc = could not find container \"e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8\": container with ID starting with e2efa8185975ea1f7efc7904e371d5fd5b99516cc540895141c41ec596383ae8 not found: ID does not exist" Oct 30 21:30:26.116581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116532 4988 scope.go:122] "RemoveContainer" containerID="60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8" Oct 30 21:30:26.116775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.116755 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8"} err="failed to get container status \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": rpc error: code = NotFound desc = could not find container \"60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8\": container with ID starting with 60b11946a864512f4ba43a2d0f32fad72f6cd688c4da3e0c10167a3c00ff86f8 not found: ID does not exist" Oct 30 21:30:26.163358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.163319 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-e73afc79-aa37-49a0-8744-633c03cad1f4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a155a55c1ce0e65b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-468025a65b88095496ffee260c47731d5a0a7d1245692f39a473b974fd0f0891\"" Oct 30 21:30:26.244935 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.244903 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:30:26.251852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:26.251826 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 21:30:26.394185 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:26.394152 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:30:27.957590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:27.957557 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:28.015023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:28.014984 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" path="/var/lib/kubelet/pods/7abef4a3-111c-4183-9aae-1836524b0fa1/volumes" Oct 30 21:30:28.063255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:28.063212 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.102:9093/-/ready\": dial tcp 10.132.0.102:9093: connect: connection refused" Oct 30 21:30:28.160316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:28.160275 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:28.330766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:28.330668 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:29.597030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:29.596994 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:29.792837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:29.792795 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:29.942599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:29.942561 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:30.099201 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:30.099161 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:33.062621 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:33.062574 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.102:9093/-/ready\": dial tcp 10.132.0.102:9093: connect: connection refused" Oct 30 21:30:35.845067 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.844999 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:30:35.967969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.967930 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.967986 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968012 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968036 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968101 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968240 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968312 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out\") pod \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\" (UID: \"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8\") " Oct 30 21:30:35.968475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968371 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:35.968624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.968604 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:35.970853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.970817 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:35.970968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.970900 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "kube-api-access-mzjw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:35.970968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.970910 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:35.970968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.970957 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:35.975067 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.975042 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:35.983258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:35.983231 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config" pod "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" (UID: "a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:36.069196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.069160 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjw6\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-kube-api-access-mzjw6\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:36.069196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.069193 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:36.069196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.069203 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:36.069473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.069212 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:36.069473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.069232 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:36.069473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.069241 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:36.130928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.130833 4988 generic.go:356] "Generic (PLEG): container finished" podID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerID="6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea" exitCode=0 Oct 30 21:30:36.130928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.130901 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:30:36.131129 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.130924 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerDied","Data":"6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea"} Oct 30 21:30:36.131129 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.130990 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8","Type":"ContainerDied","Data":"517221bdae42899f30b402c12f2b6936eaeee949dbe52fded491192a7dd1ed8d"} Oct 30 21:30:36.131129 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.131017 4988 scope.go:122] "RemoveContainer" containerID="66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1" Oct 30 21:30:36.140849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.140829 4988 scope.go:122] "RemoveContainer" containerID="6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea" Oct 30 21:30:36.148692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.148663 4988 scope.go:122] "RemoveContainer" containerID="a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76" Oct 30 21:30:36.156821 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.156796 4988 scope.go:122] "RemoveContainer" containerID="66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1" Oct 30 21:30:36.157146 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:36.157126 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1\": container with ID starting with 66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1 not found: ID does not exist" containerID="66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1" Oct 30 21:30:36.157199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.157158 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1"} err="failed to get container status \"66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1\": rpc error: code = NotFound desc = could not find container \"66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1\": container with ID starting with 66af755f4b56ce399efc987513bd1643a312580296fc58fc5754ac837c24c6b1 not found: ID does not exist" Oct 30 21:30:36.157199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.157177 4988 scope.go:122] "RemoveContainer" containerID="6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea" Oct 30 21:30:36.157446 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:36.157405 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea\": container with ID starting with 6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea not found: ID does not exist" containerID="6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea" Oct 30 21:30:36.157496 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.157452 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea"} err="failed to get container status \"6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea\": rpc error: code = NotFound desc = could not find container \"6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea\": container with ID starting with 6cdd97ae52211dfdb60a70e6a25c97353be865783fe4d42c5a4cf4043ecf4eea not found: ID does not exist" Oct 30 21:30:36.157496 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.157466 4988 scope.go:122] "RemoveContainer" containerID="a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76" Oct 30 21:30:36.157723 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:36.157707 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76\": container with ID starting with a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76 not found: ID does not exist" containerID="a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76" Oct 30 21:30:36.157763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.157727 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76"} err="failed to get container status \"a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76\": rpc error: code = NotFound desc = could not find container \"a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76\": container with ID starting with a2181a03a28004393f8d5bf865f252fea42524381c34707a558ad972aa5c9a76 not found: ID does not exist" Oct 30 21:30:36.296381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.296338 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:30:36.300296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:36.300267 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:30:38.014930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:38.014892 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" path="/var/lib/kubelet/pods/a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8/volumes" Oct 30 21:30:43.161163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:43.161127 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.332218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:43.332177 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.488376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:43.488276 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.645707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:43.645667 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.798966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:43.798880 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:43.996971 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:43.996935 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:44.169312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.169260 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:44.223206 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223170 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb"] Oct 30 21:30:44.223494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223481 4988 cpu_manager.go:414] "containerMap: removing container" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="prom-label-proxy" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223495 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="prom-label-proxy" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223514 4988 cpu_manager.go:414] "containerMap: removing container" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="kube-rbac-proxy" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223519 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="kube-rbac-proxy" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223526 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="config-reloader" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223532 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="config-reloader" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223539 4988 cpu_manager.go:414] "containerMap: removing container" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="thanos-sidecar" Oct 30 21:30:44.223543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223543 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="thanos-sidecar" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223560 4988 cpu_manager.go:414] "containerMap: removing container" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="prom-label-proxy" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223565 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="prom-label-proxy" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223572 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="init-config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223576 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="init-config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223588 4988 cpu_manager.go:414] "containerMap: removing container" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="method-gate" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223592 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="method-gate" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223597 4988 cpu_manager.go:414] "containerMap: removing container" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="method-gate" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223601 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="method-gate" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223607 4988 cpu_manager.go:414] "containerMap: removing container" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="prometheus" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223611 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="prometheus" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223618 4988 cpu_manager.go:414] "containerMap: removing container" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223622 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223627 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223631 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223638 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="alertmanager" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223642 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="alertmanager" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223652 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway-opa" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223656 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway-opa" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223662 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223666 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223673 4988 cpu_manager.go:414] "containerMap: removing container" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="kube-rbac-proxy" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223677 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="kube-rbac-proxy" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223684 4988 cpu_manager.go:414] "containerMap: removing container" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="init-config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223689 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="init-config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223738 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="prom-label-proxy" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223747 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="method-gate" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223753 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="method-gate" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223759 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="prometheus" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223763 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="config-reloader" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223768 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo" Oct 30 21:30:44.223763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223774 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="alertmanager" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223780 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway-opa" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223784 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c67e49e-c23e-4dd0-af65-e8232ff06103" containerName="tempo-gateway" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223789 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="kube-rbac-proxy" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223794 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="83237bc5-a9b6-44af-b886-44efb2a8fbd7" containerName="prom-label-proxy" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223800 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d18b72-f340-4eb2-a148-429dfbc2c7a6" containerName="kube-rbac-proxy" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223805 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ff9f2c-b555-4e4d-8d29-2020a5ef1ef8" containerName="config-reloader" Oct 30 21:30:44.224647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.223810 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abef4a3-111c-4183-9aae-1836524b0fa1" containerName="thanos-sidecar" Oct 30 21:30:44.226927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.226907 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.331405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331341 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:30:44.331405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331396 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-gk9l2\"" Oct 30 21:30:44.331674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331369 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:30:44.331674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331378 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:30:44.331674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331353 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:30:44.331674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331463 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:30:44.331674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.331472 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:30:44.332731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.332711 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb"] Oct 30 21:30:44.335910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.335892 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.335974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.335931 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.335974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.335947 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.336080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.336022 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.336080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.336071 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22mh\" (UniqueName: \"kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.337815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.337793 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4"] Oct 30 21:30:44.341360 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.341342 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.394945 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.394909 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:30:44.394945 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.394916 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:30:44.395159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.394974 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-8wshw\"" Oct 30 21:30:44.399888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.399856 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4"] Oct 30 21:30:44.436762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436665 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.436762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436706 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.436762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436752 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.437083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436770 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.437083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436785 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.437083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436812 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22mh\" (UniqueName: \"kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.437083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436839 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.437083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.436965 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n76n\" (UniqueName: \"kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.437337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.437310 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.437545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.437528 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.437602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.437567 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.439482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.439461 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.459034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.458995 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:44.501334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.501287 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22mh\" (UniqueName: \"kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.537667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.537626 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.537886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.537680 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n76n\" (UniqueName: \"kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.537886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.537717 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.538356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.538329 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.540169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.540143 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.544036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.544016 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:30:44.601115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.601091 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n76n\" (UniqueName: \"kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n\") pod \"data-science-thanos-querier-proxy-77657866bd-qlbd4\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.659240 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.659195 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:30:44.922652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:44.922615 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb"] Oct 30 21:30:44.925502 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:30:44.925477 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde57ef52_bc32_4ed6_8962_4ee46f52cace.slice/crio-995a1311ee06ff37789b37a3e3bb9e5735d3cee99535fcca4fe214092fd9e51b WatchSource:0}: Error finding container 995a1311ee06ff37789b37a3e3bb9e5735d3cee99535fcca4fe214092fd9e51b: Status 404 returned error can't find the container with id 995a1311ee06ff37789b37a3e3bb9e5735d3cee99535fcca4fe214092fd9e51b Oct 30 21:30:45.038465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.038394 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4"] Oct 30 21:30:45.041972 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:30:45.041935 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1f7324_1e4e_4c34_93a5_c29e4eb9c9de.slice/crio-e07cf4f98e277b0c34f2c49df33aca3858d6a802ac506dc597b873a38eedfdb4 WatchSource:0}: Error finding container e07cf4f98e277b0c34f2c49df33aca3858d6a802ac506dc597b873a38eedfdb4: Status 404 returned error can't find the container with id e07cf4f98e277b0c34f2c49df33aca3858d6a802ac506dc597b873a38eedfdb4 Oct 30 21:30:45.180045 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.180009 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerStarted","Data":"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45"} Oct 30 21:30:45.180532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.180119 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerStarted","Data":"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc"} Oct 30 21:30:45.180532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.180146 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerStarted","Data":"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68"} Oct 30 21:30:45.180532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.180174 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerStarted","Data":"995a1311ee06ff37789b37a3e3bb9e5735d3cee99535fcca4fe214092fd9e51b"} Oct 30 21:30:45.181547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.181520 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerStarted","Data":"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51"} Oct 30 21:30:45.181642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.181576 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerStarted","Data":"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6"} Oct 30 21:30:45.181642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.181600 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerStarted","Data":"e07cf4f98e277b0c34f2c49df33aca3858d6a802ac506dc597b873a38eedfdb4"} Oct 30 21:30:45.299518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.299446 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" podStartSLOduration=1.299404889 podStartE2EDuration="1.299404889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:45.299108184 +0000 UTC m=+3273.406604094" watchObservedRunningTime="2026-10-30 21:30:45.299404889 +0000 UTC m=+3273.406900799" Oct 30 21:30:45.842108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:45.842069 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:46.189256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:46.189215 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerStarted","Data":"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058"} Oct 30 21:30:46.308379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:46.308313 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" podStartSLOduration=2.30829591 podStartE2EDuration="2.30829591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:46.306908756 +0000 UTC m=+3274.414404666" watchObservedRunningTime="2026-10-30 21:30:46.30829591 +0000 UTC m=+3274.415791812" Oct 30 21:30:48.965991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:48.965950 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:49.117595 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:49.117553 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:49.276633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:49.276543 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:49.421698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:30:49.421664 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:30:56.394805 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:30:56.394722 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:31:01.202406 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:01.202367 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.524796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:01.524712 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.669182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:01.669118 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:01.826567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:01.826482 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.408936 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.408885 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.575381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.575340 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:02.653286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.653250 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:31:02.658158 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.658135 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.670705 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670587 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.670705 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670650 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.670939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670707 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.670939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670749 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.670939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670774 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.670939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670792 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.671175 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.670992 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sphq\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.705907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.705868 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:31:02.709686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.709665 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.754057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.754019 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-2l2zd\"" Oct 30 21:31:02.754057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.754050 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:31:02.754057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.754062 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:31:02.754316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.754017 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:31:02.754316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.754014 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:31:02.755824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.755742 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:31:02.771641 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771604 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sphq\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.771837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771650 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.771837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771680 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.771837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771699 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.771837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771722 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.772054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771834 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.772054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.771874 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.772276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.772213 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.775056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.775028 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.775056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.775047 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.775466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.775443 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.775788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.775743 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.776310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.775840 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:31:02.776310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.776004 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.804037 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.803988 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:31:02.804221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.804137 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:31:02.804221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.804216 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:31:02.811009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.810983 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:31:02.811153 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.811090 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:31:02.811194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.811155 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:31:02.811312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.811225 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:31:02.811312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.811267 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:31:02.811312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.811283 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:31:02.811449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.811355 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jq9nq\"" Oct 30 21:31:02.832481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.832448 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sphq\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:02.974017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.973921 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.973972 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974024 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974128 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974145 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974194 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974236 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974260 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974286 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sn78\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974333 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974350 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.974602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.974453 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:02.975909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:02.975895 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:03.074880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.074843 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.074880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.074881 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.074907 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.074927 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.074959 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.074981 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075013 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075037 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075067 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075095 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075125 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075162 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sn78\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075651 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.075826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075738 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.076047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.075998 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.077972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.077942 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078049 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078276 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078344 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078498 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078514 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078584 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.078686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.078670 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.136366 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.136332 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sn78\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:03.407321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:03.407286 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:31:03.410730 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:31:03.410693 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f80082b_df45_4ce2_b602_11dc60308365.slice/crio-40a4e10a550656bbe51a1ee8cd24c60ff69d07795802040c1bde49da1bbd878e WatchSource:0}: Error finding container 40a4e10a550656bbe51a1ee8cd24c60ff69d07795802040c1bde49da1bbd878e: Status 404 returned error can't find the container with id 40a4e10a550656bbe51a1ee8cd24c60ff69d07795802040c1bde49da1bbd878e Oct 30 21:31:04.272863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:04.272820 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerStarted","Data":"6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4"} Oct 30 21:31:04.273020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:04.272890 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerStarted","Data":"40a4e10a550656bbe51a1ee8cd24c60ff69d07795802040c1bde49da1bbd878e"} Oct 30 21:31:07.288503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:07.288468 4988 generic.go:356] "Generic (PLEG): container finished" podID="9f80082b-df45-4ce2-b602-11dc60308365" containerID="6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4" exitCode=0 Oct 30 21:31:07.288939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:07.288538 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerDied","Data":"6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4"} Oct 30 21:31:08.294793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:08.294704 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerStarted","Data":"0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591"} Oct 30 21:31:08.294793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:08.294771 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerStarted","Data":"62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41"} Oct 30 21:31:08.295236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:08.294956 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:08.506577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:08.506512 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.221846343 podStartE2EDuration="6.506497046s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284650703s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:08.504751707 +0000 UTC m=+3296.612247616" watchObservedRunningTime="2026-10-30 21:31:08.506497046 +0000 UTC m=+3296.613992956" Oct 30 21:31:10.128918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.128884 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.233380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.233346 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.329763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.329723 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.329959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.329865 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.381985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.381906 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.520915 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.520883 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.590243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.590203 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:10.829353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:10.829315 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:11.314989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:11.314957 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:31:11.317869 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:31:11.317844 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62c3d5f_07d2_4f26_aed1_c2cc79e2a0ab.slice/crio-7bb428140bc9b6d86df70b04e36f86f863e4318253355c98d534c285ad63334f WatchSource:0}: Error finding container 7bb428140bc9b6d86df70b04e36f86f863e4318253355c98d534c285ad63334f: Status 404 returned error can't find the container with id 7bb428140bc9b6d86df70b04e36f86f863e4318253355c98d534c285ad63334f Oct 30 21:31:12.313946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:12.313820 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerStarted","Data":"dcd46ed746d63bd441ec5ec94abb6f16f2a1603dcf47da42d704c2bd9b5a6f4e"} Oct 30 21:31:12.313946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:12.313891 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerStarted","Data":"7bb428140bc9b6d86df70b04e36f86f863e4318253355c98d534c285ad63334f"} Oct 30 21:31:12.337087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:12.337053 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:31:12.344471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:12.344445 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:31:14.300719 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:14.300682 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:31:15.327495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:15.327464 4988 generic.go:356] "Generic (PLEG): container finished" podID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerID="dcd46ed746d63bd441ec5ec94abb6f16f2a1603dcf47da42d704c2bd9b5a6f4e" exitCode=0 Oct 30 21:31:15.327908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:15.327547 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerDied","Data":"dcd46ed746d63bd441ec5ec94abb6f16f2a1603dcf47da42d704c2bd9b5a6f4e"} Oct 30 21:31:16.334862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:16.334769 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerStarted","Data":"ed2c60490fc742036e8d9ee38d9d6c3f83f478770fad92db4f1defd5e9cd9807"} Oct 30 21:31:16.334862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:16.334841 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerStarted","Data":"2dbd893f752bb8af6be45a1ea8b8a36e6f5ab363d6493aa164dc29b8e015d46c"} Oct 30 21:31:16.335332 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:16.334872 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerStarted","Data":"cfdcc6791fd8b7fcee917f3f82f6a947a624cf45a1e181837f4f7ffddf6be450"} Oct 30 21:31:16.463342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:16.463277 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=11.379015485 podStartE2EDuration="14.463263102s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.084247617s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:16.462815779 +0000 UTC m=+3304.570311705" watchObservedRunningTime="2026-10-30 21:31:16.463263102 +0000 UTC m=+3304.570759012" Oct 30 21:31:20.271079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:20.271044 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:20.830372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:20.830333 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:21.610791 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:21.610749 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:21.774204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:21.774166 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:21.965140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:21.965100 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:25.830324 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:25.830286 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:25.835932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:25.835904 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:26.390921 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:26.390893 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:31:26.397107 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:31:26.397076 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:31:27.582681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:27.582647 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:27.784528 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:27.784487 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:27.999890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:27.999849 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:28.190800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:28.190764 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:42.734530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:42.734484 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:48.652647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:48.652605 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:48.829265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:48.829231 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:49.026006 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:49.025909 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:49.561506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:49.561472 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.674719 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:54.674682 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:54.894559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:54.894520 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:55.726127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:55.726086 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:55.908594 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:55.908563 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:56.355998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:31:56.355962 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:31:56.395908 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:31:56.395877 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:32:00.590509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:00.590469 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:00.748447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:00.748392 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.688405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:01.688364 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:01.881510 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:01.881472 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:02.544767 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:02.544730 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:02.747983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:02.747939 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:03.169004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:03.168964 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.552353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:06.552317 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:06.705345 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:06.705283 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:07.303007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:07.302964 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:07.447074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:07.447039 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:07.598819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:07.598726 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:12.404966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:12.404926 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:12.555180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:12.555138 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.686389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:13.686351 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:13.880874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:13.880838 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:14.029954 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:14.029866 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:18.274150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:18.274109 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:18.519313 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:18.519274 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.148424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:20.148351 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.350440 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:20.350372 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.512088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:20.512000 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:20.667704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:20.667669 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.204120 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:24.204029 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.347199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:24.347160 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.498610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:24.498526 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.756696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:24.756609 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:24.912214 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:24.912181 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:26.395383 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:26.395345 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:32:29.945707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:29.945655 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:30.101718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:30.101678 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:42.140913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.140871 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb"] Oct 30 21:32:42.141459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.141192 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="kube-rbac-proxy" containerID="cri-o://3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" gracePeriod=30 Oct 30 21:32:42.141459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.141257 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="method-gate" containerID="cri-o://8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" gracePeriod=30 Oct 30 21:32:42.141459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.141255 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="prom-label-proxy" containerID="cri-o://81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" gracePeriod=30 Oct 30 21:32:42.247336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.247296 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4"] Oct 30 21:32:42.247683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.247649 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="method-gate" containerID="cri-o://22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" gracePeriod=30 Oct 30 21:32:42.247793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.247653 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="prom-label-proxy" containerID="cri-o://c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" gracePeriod=30 Oct 30 21:32:42.247793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.247646 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="kube-rbac-proxy" containerID="cri-o://02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" gracePeriod=30 Oct 30 21:32:42.384105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.384065 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:32:42.442955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.442917 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca\") pod \"de57ef52-bc32-4ed6-8962-4ee46f52cace\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " Oct 30 21:32:42.443132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.442984 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca\") pod \"de57ef52-bc32-4ed6-8962-4ee46f52cace\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " Oct 30 21:32:42.443132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443010 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls\" (UniqueName: \"kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls\") pod \"de57ef52-bc32-4ed6-8962-4ee46f52cace\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " Oct 30 21:32:42.443132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443081 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh\" (UniqueName: \"kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh\") pod \"de57ef52-bc32-4ed6-8962-4ee46f52cace\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " Oct 30 21:32:42.443292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443247 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config\") pod \"de57ef52-bc32-4ed6-8962-4ee46f52cace\" (UID: \"de57ef52-bc32-4ed6-8962-4ee46f52cace\") " Oct 30 21:32:42.443398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443373 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca" pod "de57ef52-bc32-4ed6-8962-4ee46f52cace" (UID: "de57ef52-bc32-4ed6-8962-4ee46f52cace"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:42.443505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443485 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca" pod "de57ef52-bc32-4ed6-8962-4ee46f52cace" (UID: "de57ef52-bc32-4ed6-8962-4ee46f52cace"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:42.443712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443696 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.443790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443718 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.443790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.443776 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config" pod "de57ef52-bc32-4ed6-8962-4ee46f52cace" (UID: "de57ef52-bc32-4ed6-8962-4ee46f52cace"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:42.445733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.445710 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls" pod "de57ef52-bc32-4ed6-8962-4ee46f52cace" (UID: "de57ef52-bc32-4ed6-8962-4ee46f52cace"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:42.445812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.445731 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh" pod "de57ef52-bc32-4ed6-8962-4ee46f52cace" (UID: "de57ef52-bc32-4ed6-8962-4ee46f52cace"). InnerVolumeSpecName "kube-api-access-p22mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:42.493754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.493726 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:32:42.544832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.544796 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/de57ef52-bc32-4ed6-8962-4ee46f52cace-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.544996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.544836 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22mh\" (UniqueName: \"kubernetes.io/projected/de57ef52-bc32-4ed6-8962-4ee46f52cace-kube-api-access-p22mh\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.544996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.544856 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de57ef52-bc32-4ed6-8962-4ee46f52cace-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.645948 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.645919 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config\" (UniqueName: \"kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config\") pod \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " Oct 30 21:32:42.646115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.645961 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n\" (UniqueName: \"kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n\") pod \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " Oct 30 21:32:42.646115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.645999 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls\" (UniqueName: \"kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls\") pod \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\" (UID: \"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de\") " Oct 30 21:32:42.646310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.646288 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config" pod "6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" (UID: "6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:42.648025 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.648000 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls" pod "6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" (UID: "6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:42.648145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.648072 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n" pod "6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" (UID: "6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de"). InnerVolumeSpecName "kube-api-access-5n76n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:42.731456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731360 4988 generic.go:356] "Generic (PLEG): container finished" podID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerID="81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" exitCode=0 Oct 30 21:32:42.731456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731388 4988 generic.go:356] "Generic (PLEG): container finished" podID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerID="8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" exitCode=0 Oct 30 21:32:42.731456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731395 4988 generic.go:356] "Generic (PLEG): container finished" podID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerID="3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" exitCode=0 Oct 30 21:32:42.731456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731406 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" Oct 30 21:32:42.731456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731442 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerDied","Data":"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45"} Oct 30 21:32:42.731832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731516 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerDied","Data":"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc"} Oct 30 21:32:42.731832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731552 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerDied","Data":"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68"} Oct 30 21:32:42.731832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731579 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb" event={"ID":"de57ef52-bc32-4ed6-8962-4ee46f52cace","Type":"ContainerDied","Data":"995a1311ee06ff37789b37a3e3bb9e5735d3cee99535fcca4fe214092fd9e51b"} Oct 30 21:32:42.731832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.731589 4988 scope.go:122] "RemoveContainer" containerID="81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" Oct 30 21:32:42.733591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733561 4988 generic.go:356] "Generic (PLEG): container finished" podID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerID="c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" exitCode=0 Oct 30 21:32:42.733591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733589 4988 generic.go:356] "Generic (PLEG): container finished" podID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerID="22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" exitCode=0 Oct 30 21:32:42.733748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733601 4988 generic.go:356] "Generic (PLEG): container finished" podID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerID="02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" exitCode=0 Oct 30 21:32:42.733748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733639 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" Oct 30 21:32:42.733748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733642 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerDied","Data":"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058"} Oct 30 21:32:42.733748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733715 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerDied","Data":"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51"} Oct 30 21:32:42.733903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733748 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerDied","Data":"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6"} Oct 30 21:32:42.733903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.733774 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4" event={"ID":"6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de","Type":"ContainerDied","Data":"e07cf4f98e277b0c34f2c49df33aca3858d6a802ac506dc597b873a38eedfdb4"} Oct 30 21:32:42.742424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.742392 4988 scope.go:122] "RemoveContainer" containerID="8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" Oct 30 21:32:42.747196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.747175 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.747293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.747199 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n76n\" (UniqueName: \"kubernetes.io/projected/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-kube-api-access-5n76n\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.747293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.747209 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:42.750251 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.750223 4988 scope.go:122] "RemoveContainer" containerID="3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" Oct 30 21:32:42.758646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.758623 4988 scope.go:122] "RemoveContainer" containerID="81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" Oct 30 21:32:42.758971 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:42.758950 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": container with ID starting with 81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45 not found: ID does not exist" containerID="81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" Oct 30 21:32:42.759009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.758984 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45"} err="failed to get container status \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": rpc error: code = NotFound desc = could not find container \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": container with ID starting with 81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45 not found: ID does not exist" Oct 30 21:32:42.759009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759003 4988 scope.go:122] "RemoveContainer" containerID="8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" Oct 30 21:32:42.759291 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:42.759272 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": container with ID starting with 8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc not found: ID does not exist" containerID="8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" Oct 30 21:32:42.759335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759301 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc"} err="failed to get container status \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": rpc error: code = NotFound desc = could not find container \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": container with ID starting with 8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc not found: ID does not exist" Oct 30 21:32:42.759335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759317 4988 scope.go:122] "RemoveContainer" containerID="3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" Oct 30 21:32:42.759565 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:42.759547 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": container with ID starting with 3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68 not found: ID does not exist" containerID="3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" Oct 30 21:32:42.759620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759571 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68"} err="failed to get container status \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": rpc error: code = NotFound desc = could not find container \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": container with ID starting with 3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68 not found: ID does not exist" Oct 30 21:32:42.759620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759586 4988 scope.go:122] "RemoveContainer" containerID="81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" Oct 30 21:32:42.759809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759790 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45"} err="failed to get container status \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": rpc error: code = NotFound desc = could not find container \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": container with ID starting with 81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45 not found: ID does not exist" Oct 30 21:32:42.759853 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759810 4988 scope.go:122] "RemoveContainer" containerID="8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" Oct 30 21:32:42.760008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.759991 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc"} err="failed to get container status \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": rpc error: code = NotFound desc = could not find container \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": container with ID starting with 8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc not found: ID does not exist" Oct 30 21:32:42.760043 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760010 4988 scope.go:122] "RemoveContainer" containerID="3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" Oct 30 21:32:42.760226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760207 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68"} err="failed to get container status \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": rpc error: code = NotFound desc = could not find container \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": container with ID starting with 3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68 not found: ID does not exist" Oct 30 21:32:42.760272 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760227 4988 scope.go:122] "RemoveContainer" containerID="81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45" Oct 30 21:32:42.760442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760398 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45"} err="failed to get container status \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": rpc error: code = NotFound desc = could not find container \"81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45\": container with ID starting with 81456ff5d3baf947f9d4bf5df91e703db544136918c4b8738dcfb4c0e4a97d45 not found: ID does not exist" Oct 30 21:32:42.760495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760444 4988 scope.go:122] "RemoveContainer" containerID="8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc" Oct 30 21:32:42.760701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760685 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc"} err="failed to get container status \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": rpc error: code = NotFound desc = could not find container \"8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc\": container with ID starting with 8d84d19a9c2d24849289a350aa9cea1114cc989249c1dc392c65827f0a8fb0cc not found: ID does not exist" Oct 30 21:32:42.760771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760700 4988 scope.go:122] "RemoveContainer" containerID="3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68" Oct 30 21:32:42.760936 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760921 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68"} err="failed to get container status \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": rpc error: code = NotFound desc = could not find container \"3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68\": container with ID starting with 3418c2a51d1e11ef9bd9a80ccb6b03ba72a123b7d557870ca2b290653f810d68 not found: ID does not exist" Oct 30 21:32:42.760985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.760936 4988 scope.go:122] "RemoveContainer" containerID="c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" Oct 30 21:32:42.770793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.770767 4988 scope.go:122] "RemoveContainer" containerID="22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" Oct 30 21:32:42.779353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.779330 4988 scope.go:122] "RemoveContainer" containerID="02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" Oct 30 21:32:42.787842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.787822 4988 scope.go:122] "RemoveContainer" containerID="c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" Oct 30 21:32:42.788151 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:42.788131 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": container with ID starting with c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058 not found: ID does not exist" containerID="c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" Oct 30 21:32:42.788202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788161 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058"} err="failed to get container status \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": rpc error: code = NotFound desc = could not find container \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": container with ID starting with c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058 not found: ID does not exist" Oct 30 21:32:42.788202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788181 4988 scope.go:122] "RemoveContainer" containerID="22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" Oct 30 21:32:42.788460 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:42.788442 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": container with ID starting with 22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51 not found: ID does not exist" containerID="22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" Oct 30 21:32:42.788507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788467 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51"} err="failed to get container status \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": rpc error: code = NotFound desc = could not find container \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": container with ID starting with 22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51 not found: ID does not exist" Oct 30 21:32:42.788507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788481 4988 scope.go:122] "RemoveContainer" containerID="02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" Oct 30 21:32:42.788681 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:42.788666 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": container with ID starting with 02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6 not found: ID does not exist" containerID="02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" Oct 30 21:32:42.788723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788685 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6"} err="failed to get container status \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": rpc error: code = NotFound desc = could not find container \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": container with ID starting with 02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6 not found: ID does not exist" Oct 30 21:32:42.788723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788696 4988 scope.go:122] "RemoveContainer" containerID="c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" Oct 30 21:32:42.788897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788881 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058"} err="failed to get container status \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": rpc error: code = NotFound desc = could not find container \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": container with ID starting with c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058 not found: ID does not exist" Oct 30 21:32:42.788939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.788897 4988 scope.go:122] "RemoveContainer" containerID="22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" Oct 30 21:32:42.789071 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789057 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51"} err="failed to get container status \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": rpc error: code = NotFound desc = could not find container \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": container with ID starting with 22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51 not found: ID does not exist" Oct 30 21:32:42.789164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789072 4988 scope.go:122] "RemoveContainer" containerID="02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" Oct 30 21:32:42.789316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789301 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6"} err="failed to get container status \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": rpc error: code = NotFound desc = could not find container \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": container with ID starting with 02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6 not found: ID does not exist" Oct 30 21:32:42.789316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789315 4988 scope.go:122] "RemoveContainer" containerID="c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058" Oct 30 21:32:42.789535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789514 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058"} err="failed to get container status \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": rpc error: code = NotFound desc = could not find container \"c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058\": container with ID starting with c3d29f8a1e8eb6d02de5f61c64bac07c13d7a00dd3d51fc18ecebc0e6a45b058 not found: ID does not exist" Oct 30 21:32:42.789599 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789535 4988 scope.go:122] "RemoveContainer" containerID="22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51" Oct 30 21:32:42.789758 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789743 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51"} err="failed to get container status \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": rpc error: code = NotFound desc = could not find container \"22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51\": container with ID starting with 22d3872093be3269b4e065da8f65ca47c221da7426855de788d75f13ac26ce51 not found: ID does not exist" Oct 30 21:32:42.789800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789758 4988 scope.go:122] "RemoveContainer" containerID="02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6" Oct 30 21:32:42.789939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.789925 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6"} err="failed to get container status \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": rpc error: code = NotFound desc = could not find container \"02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6\": container with ID starting with 02b569173fcd54a766fbea791e4792d16668754d8a89142f8c8e90d0350d37c6 not found: ID does not exist" Oct 30 21:32:42.897207 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.897158 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb"] Oct 30 21:32:42.901091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:42.901053 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-hzdkb"] Oct 30 21:32:43.011331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.011252 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4"] Oct 30 21:32:43.015902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.015871 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-qlbd4"] Oct 30 21:32:43.194046 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.193993 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:43.389792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.389755 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:43.478578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.478539 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:32:43.478813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.478787 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="alertmanager" containerID="cri-o://62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41" gracePeriod=120 Oct 30 21:32:43.478873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.478819 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="config-reloader" containerID="cri-o://0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591" gracePeriod=120 Oct 30 21:32:43.488386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.488348 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:32:43.488705 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.488663 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="prometheus" containerID="cri-o://cfdcc6791fd8b7fcee917f3f82f6a947a624cf45a1e181837f4f7ffddf6be450" gracePeriod=600 Oct 30 21:32:43.488791 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.488762 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="config-reloader" containerID="cri-o://2dbd893f752bb8af6be45a1ea8b8a36e6f5ab363d6493aa164dc29b8e015d46c" gracePeriod=600 Oct 30 21:32:43.488863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.488762 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="thanos-sidecar" containerID="cri-o://ed2c60490fc742036e8d9ee38d9d6c3f83f478770fad92db4f1defd5e9cd9807" gracePeriod=600 Oct 30 21:32:43.554947 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.554903 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:43.742996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.742957 4988 generic.go:356] "Generic (PLEG): container finished" podID="9f80082b-df45-4ce2-b602-11dc60308365" containerID="0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591" exitCode=0 Oct 30 21:32:43.743200 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.743026 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerDied","Data":"0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591"} Oct 30 21:32:43.747855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.747823 4988 generic.go:356] "Generic (PLEG): container finished" podID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerID="ed2c60490fc742036e8d9ee38d9d6c3f83f478770fad92db4f1defd5e9cd9807" exitCode=0 Oct 30 21:32:43.747855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.747848 4988 generic.go:356] "Generic (PLEG): container finished" podID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerID="2dbd893f752bb8af6be45a1ea8b8a36e6f5ab363d6493aa164dc29b8e015d46c" exitCode=0 Oct 30 21:32:43.747855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.747854 4988 generic.go:356] "Generic (PLEG): container finished" podID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerID="cfdcc6791fd8b7fcee917f3f82f6a947a624cf45a1e181837f4f7ffddf6be450" exitCode=0 Oct 30 21:32:43.748080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.747875 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerDied","Data":"ed2c60490fc742036e8d9ee38d9d6c3f83f478770fad92db4f1defd5e9cd9807"} Oct 30 21:32:43.748080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.747935 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerDied","Data":"2dbd893f752bb8af6be45a1ea8b8a36e6f5ab363d6493aa164dc29b8e015d46c"} Oct 30 21:32:43.748080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.747971 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerDied","Data":"cfdcc6791fd8b7fcee917f3f82f6a947a624cf45a1e181837f4f7ffddf6be450"} Oct 30 21:32:43.815935 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.815900 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:43.822220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.822190 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:32:43.961472 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.961365 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.961896 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.961772 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.962616 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.962592 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.962755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.962644 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.962755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.962686 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.962755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.962725 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.962918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.962763 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.962918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.962795 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.963440 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.963325 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.967189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.963972 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:43.967189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.964380 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:43.967189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.965738 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:43.967189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.966238 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:43.967189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.966622 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:43.967653 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.967628 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.967813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.967683 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.967813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.967725 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.967813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.967785 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\" (UID: \"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab\") " Oct 30 21:32:43.968058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.967836 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:43.968142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968068 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:43.968142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968069 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:43.968142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968091 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:43.968142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968107 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:43.968142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968123 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:43.968142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968139 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:43.968537 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968155 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:43.968537 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968499 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:43.969013 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.968986 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "kube-api-access-2sn78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:43.970542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.970512 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:43.970746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.970725 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:43.970833 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.970783 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:43.973234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.973213 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:32:43.979672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:43.979641 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config" pod "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" (UID: "a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:44.016115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.016074 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" path="/var/lib/kubelet/pods/6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de/volumes" Oct 30 21:32:44.016637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.016620 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" path="/var/lib/kubelet/pods/de57ef52-bc32-4ed6-8962-4ee46f52cace/volumes" Oct 30 21:32:44.069562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069515 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sn78\" (UniqueName: \"kubernetes.io/projected/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-kube-api-access-2sn78\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.069562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069550 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.069562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069561 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.069796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069589 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:32:44.069796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069599 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.069796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069608 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.069796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.069617 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.081740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.081717 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:32:44.170739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.170695 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:32:44.300378 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.300282 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.108:9093/-/ready\": dial tcp 10.132.0.108:9093: connect: connection refused" Oct 30 21:32:44.755090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.755056 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab","Type":"ContainerDied","Data":"7bb428140bc9b6d86df70b04e36f86f863e4318253355c98d534c285ad63334f"} Oct 30 21:32:44.755276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.755118 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:32:44.755276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.755127 4988 scope.go:122] "RemoveContainer" containerID="ed2c60490fc742036e8d9ee38d9d6c3f83f478770fad92db4f1defd5e9cd9807" Oct 30 21:32:44.763592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.763522 4988 scope.go:122] "RemoveContainer" containerID="2dbd893f752bb8af6be45a1ea8b8a36e6f5ab363d6493aa164dc29b8e015d46c" Oct 30 21:32:44.771724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.771701 4988 scope.go:122] "RemoveContainer" containerID="cfdcc6791fd8b7fcee917f3f82f6a947a624cf45a1e181837f4f7ffddf6be450" Oct 30 21:32:44.780275 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.780255 4988 scope.go:122] "RemoveContainer" containerID="dcd46ed746d63bd441ec5ec94abb6f16f2a1603dcf47da42d704c2bd9b5a6f4e" Oct 30 21:32:44.925798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.925763 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:32:44.975406 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:44.975376 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:32:45.845912 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:45.845873 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:45.991538 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:45.991502 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:46.015679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:46.015645 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" path="/var/lib/kubelet/pods/a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab/volumes" Oct 30 21:32:46.290680 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:46.290642 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:47.471215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:47.471175 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:47.672473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:47.672438 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:47.821500 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:47.821379 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:47.968822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:47.968788 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:49.300964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:49.300917 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.108:9093/-/ready\": dial tcp 10.132.0.108:9093: connect: connection refused" Oct 30 21:32:49.743682 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:49.743643 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:53.728693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.728666 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:32:53.796399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.796309 4988 generic.go:356] "Generic (PLEG): container finished" podID="9f80082b-df45-4ce2-b602-11dc60308365" containerID="62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41" exitCode=0 Oct 30 21:32:53.796399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.796373 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerDied","Data":"62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41"} Oct 30 21:32:53.796399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.796379 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:32:53.796711 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.796436 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9f80082b-df45-4ce2-b602-11dc60308365","Type":"ContainerDied","Data":"40a4e10a550656bbe51a1ee8cd24c60ff69d07795802040c1bde49da1bbd878e"} Oct 30 21:32:53.796711 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.796465 4988 scope.go:122] "RemoveContainer" containerID="0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591" Oct 30 21:32:53.804737 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.804712 4988 scope.go:122] "RemoveContainer" containerID="62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41" Oct 30 21:32:53.815254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.815219 4988 scope.go:122] "RemoveContainer" containerID="6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4" Oct 30 21:32:53.823744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.823715 4988 scope.go:122] "RemoveContainer" containerID="0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591" Oct 30 21:32:53.824075 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:53.824055 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591\": container with ID starting with 0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591 not found: ID does not exist" containerID="0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591" Oct 30 21:32:53.824143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.824088 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591"} err="failed to get container status \"0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591\": rpc error: code = NotFound desc = could not find container \"0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591\": container with ID starting with 0e3c7feab281a6fc65a50d9ad9c787aef5cfd6ca2a344e73eeeaf8443dfbc591 not found: ID does not exist" Oct 30 21:32:53.824143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.824109 4988 scope.go:122] "RemoveContainer" containerID="62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41" Oct 30 21:32:53.824358 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:53.824341 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41\": container with ID starting with 62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41 not found: ID does not exist" containerID="62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41" Oct 30 21:32:53.824460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.824366 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41"} err="failed to get container status \"62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41\": rpc error: code = NotFound desc = could not find container \"62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41\": container with ID starting with 62f68ded2650851dda18c37ec65780be60e3281b5a3cdf8f24fd38cfb9a96d41 not found: ID does not exist" Oct 30 21:32:53.824460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.824381 4988 scope.go:122] "RemoveContainer" containerID="6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4" Oct 30 21:32:53.824647 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:53.824632 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4\": container with ID starting with 6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4 not found: ID does not exist" containerID="6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4" Oct 30 21:32:53.824703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.824649 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4"} err="failed to get container status \"6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4\": rpc error: code = NotFound desc = could not find container \"6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4\": container with ID starting with 6548831be231e72fda4417497f44226b375e0f9ff73229fd2cf7ce5363c2b9f4 not found: ID does not exist" Oct 30 21:32:53.854004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.853964 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854011 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854030 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854067 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854146 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854170 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854199 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume\") pod \"9f80082b-df45-4ce2-b602-11dc60308365\" (UID: \"9f80082b-df45-4ce2-b602-11dc60308365\") " Oct 30 21:32:53.854582 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.854551 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:53.856281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.856250 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:53.856547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.856511 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:53.856634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.856597 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:53.856806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.856792 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "kube-api-access-8sphq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:53.861233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.861211 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:53.867765 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.867738 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config" pod "9f80082b-df45-4ce2-b602-11dc60308365" (UID: "9f80082b-df45-4ce2-b602-11dc60308365"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:53.955424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955385 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:53.955424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955428 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f80082b-df45-4ce2-b602-11dc60308365-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:53.955424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955438 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:53.955747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955447 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:53.955747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955456 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:53.955747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955464 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f80082b-df45-4ce2-b602-11dc60308365-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:53.955747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:53.955473 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sphq\" (UniqueName: \"kubernetes.io/projected/9f80082b-df45-4ce2-b602-11dc60308365-kube-api-access-8sphq\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:54.211717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:54.211681 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:32:54.217063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:54.217033 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:32:56.015617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:56.015572 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f80082b-df45-4ce2-b602-11dc60308365" path="/var/lib/kubelet/pods/9f80082b-df45-4ce2-b602-11dc60308365/volumes" Oct 30 21:32:56.395231 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:32:56.395186 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:32:57.255994 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:57.255953 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:32:58.647635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:32:58.647589 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:04.610482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:04.610440 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:04.805679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:04.805637 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:12.570042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:12.569994 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:12.714264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:12.714227 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:12.868713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:12.868628 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.058332 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.058298 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.205445 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.205394 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.350656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.350608 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.499333 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.499244 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.646095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.646059 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.799843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.799748 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:13.955139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:13.955105 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:14.102886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.102795 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:14.339836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.339804 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8"] Oct 30 21:33:14.340188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340169 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="init-config-reloader" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340192 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="init-config-reloader" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340209 4988 cpu_manager.go:414] "containerMap: removing container" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="method-gate" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340217 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="method-gate" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340243 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="method-gate" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340251 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="method-gate" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340263 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="prom-label-proxy" Oct 30 21:33:14.340280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340270 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="prom-label-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340286 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340294 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340303 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340310 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340321 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="thanos-sidecar" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340328 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="thanos-sidecar" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340341 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="alertmanager" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340349 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="alertmanager" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340365 4988 cpu_manager.go:414] "containerMap: removing container" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="kube-rbac-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340372 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="kube-rbac-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340381 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="init-config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340389 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="init-config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340399 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="prometheus" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340405 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="prometheus" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340432 4988 cpu_manager.go:414] "containerMap: removing container" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="prom-label-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340439 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="prom-label-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340454 4988 cpu_manager.go:414] "containerMap: removing container" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="kube-rbac-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340461 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="kube-rbac-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340549 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="kube-rbac-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340565 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="method-gate" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340576 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="method-gate" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340585 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1f7324-1e4e-4c34-93a5-c29e4eb9c9de" containerName="prom-label-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340594 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340601 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="config-reloader" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340610 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="thanos-sidecar" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340618 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f80082b-df45-4ce2-b602-11dc60308365" containerName="alertmanager" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340627 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="kube-rbac-proxy" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340635 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62c3d5f-07d2-4f26-aed1-c2cc79e2a0ab" containerName="prometheus" Oct 30 21:33:14.340657 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.340644 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="de57ef52-bc32-4ed6-8962-4ee46f52cace" containerName="prom-label-proxy" Oct 30 21:33:14.343953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.343932 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.358042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.357971 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:14.390175 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.390133 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8"] Oct 30 21:33:14.410879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.410840 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.410884 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.410977 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.411066 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.411098 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.411116 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411235 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.411138 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx65f\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.411268 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.411241 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.438074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438009 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 30 21:33:14.438074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438025 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 30 21:33:14.438074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438060 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:33:14.438355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438086 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 30 21:33:14.438355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438095 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 21:33:14.438355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438108 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:33:14.438355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438174 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-sqm6h\"" Oct 30 21:33:14.438355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438225 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 30 21:33:14.438355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.438251 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 21:33:14.512552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512499 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.512552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512557 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.512808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512585 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.512808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512609 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx65f\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.512808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512636 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.512808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512779 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.513011 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512829 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.513011 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.512870 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.513494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.513471 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.513650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.513574 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.513650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.513606 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.515711 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.515676 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.515822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.515676 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.515822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.515733 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.515822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.515732 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.573825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.573789 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx65f\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f\") pod \"data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:14.601243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.601200 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:14.658049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:14.658011 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:15.086296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.086216 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8"] Oct 30 21:33:15.089662 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:33:15.089633 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1931617_7e55_450f_9c2d_2e058fc3b23e.slice/crio-d26dfab102996958e3035cbd99fadc852d548e8b4e9a4f972687099790a2733c WatchSource:0}: Error finding container d26dfab102996958e3035cbd99fadc852d548e8b4e9a4f972687099790a2733c: Status 404 returned error can't find the container with id d26dfab102996958e3035cbd99fadc852d548e8b4e9a4f972687099790a2733c Oct 30 21:33:15.091562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.091545 4988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:33:15.290015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.289975 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.389836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.389804 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:33:15.394234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.394215 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.440744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.440703 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.447274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.447247 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-usage-logs-loki-secret\"" Oct 30 21:33:15.447274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.447253 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 30 21:33:15.447274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.447268 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 30 21:33:15.447563 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.447284 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-g2x6b\"" Oct 30 21:33:15.447563 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.447347 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 30 21:33:15.454029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.453997 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:33:15.522199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.522161 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.522377 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.522208 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.522377 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.522284 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.522377 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.522349 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.522568 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.522430 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.522568 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.522533 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfmd\" (UniqueName: \"kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.590359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.590316 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:15.623368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.623327 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.623606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.623387 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.623606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.623445 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.623606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.623506 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.623606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.623530 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.623606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.623574 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfmd\" (UniqueName: \"kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.624289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.624259 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.624444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.624333 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.625917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.625885 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.626024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.626005 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.626152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.626137 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.685104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.685031 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfmd\" (UniqueName: \"kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:15.889347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.889310 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" event={"ID":"a1931617-7e55-450f-9c2d-2e058fc3b23e","Type":"ContainerStarted","Data":"d26dfab102996958e3035cbd99fadc852d548e8b4e9a4f972687099790a2733c"} Oct 30 21:33:15.948566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.948473 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 21:33:15.952906 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:15.952876 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.005547 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.005509 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-http\"" Oct 30 21:33:16.005938 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.005916 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-grpc\"" Oct 30 21:33:16.015344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.015314 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 21:33:16.056083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.056049 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:16.260768 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.260679 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:16.329502 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.329460 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.329693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.329519 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.329693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.329669 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.329799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.329718 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.329799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.329777 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.329910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.329874 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxhw\" (UniqueName: \"kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.430424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.430377 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.430636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.430473 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.430636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.430525 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.430636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.430571 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.430636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.430595 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.430636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.430624 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxhw\" (UniqueName: \"kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.431281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.431254 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.431503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.431480 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.433101 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.433075 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.433260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.433153 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.433372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.433351 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.492688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.492655 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxhw\" (UniqueName: \"kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:16.963427 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:16.963375 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:20.913424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:20.913378 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" event={"ID":"a1931617-7e55-450f-9c2d-2e058fc3b23e","Type":"ContainerStarted","Data":"fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1"} Oct 30 21:33:20.944989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:20.944950 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:23.090339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.090295 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.090339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.090342 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.090839 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.090363 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.244968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.244928 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.244968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.244961 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.245213 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.244995 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.291931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.291885 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.291931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.291936 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.292223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.291962 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.292223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.292035 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.292223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.292105 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.292223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.292105 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.344373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.344300 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") DevicePath \"csi-155d7e76a00fba862a255a008b4646e5e30bae03a1ce3c419eed77bd55a26b19\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.344503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.344303 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") DevicePath \"csi-df2bbbe12036b89ff6a72b070aa5ff076d9a9f83f6e13ab3c18c3d8609fd3461\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.344503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.344313 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") DevicePath \"csi-2c615d72aa7032fc0973683fa68549afa65f559f301223e7d8c7fbe6ace7eca0\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.607914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.607879 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3599031f1f369668381f8de34b2d2208c927c911462778ffe2e7b80ddd4e9f9a/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.608092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.607904 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c029c8b63da86bb80198b1e1499e34b65ec9380ac44499e9974f5cbb2b19b1a0/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.626819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.626787 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fdd263e80916c99f6ceb9f70acac6dacd6dc4d7bb485176bf2f162ed785207a5/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.672059 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.672026 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.673630 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.673604 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.689713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.689674 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:23.764571 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.764528 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:23.805990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:23.805956 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:24.190594 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:24.190561 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:33:24.192695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:24.192676 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 21:33:24.193710 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:33:24.193686 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8db61d_594f_4273_b9bf_228e1f011484.slice/crio-e4d41b53f18a786fe31deab81bb2bbbc3063ca53f7a9cf9574c4367072c21b72 WatchSource:0}: Error finding container e4d41b53f18a786fe31deab81bb2bbbc3063ca53f7a9cf9574c4367072c21b72: Status 404 returned error can't find the container with id e4d41b53f18a786fe31deab81bb2bbbc3063ca53f7a9cf9574c4367072c21b72 Oct 30 21:33:24.231802 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:33:24.231766 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b7513df_1500_4aa5_9f16_d20fc090fbc9.slice/crio-29a4199b54cea3d4ed33f0c0c80acfdea6e88446d56ed0a42184f6be1bab7962 WatchSource:0}: Error finding container 29a4199b54cea3d4ed33f0c0c80acfdea6e88446d56ed0a42184f6be1bab7962: Status 404 returned error can't find the container with id 29a4199b54cea3d4ed33f0c0c80acfdea6e88446d56ed0a42184f6be1bab7962 Oct 30 21:33:24.936515 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:24.936468 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"4b7513df-1500-4aa5-9f16-d20fc090fbc9","Type":"ContainerStarted","Data":"29a4199b54cea3d4ed33f0c0c80acfdea6e88446d56ed0a42184f6be1bab7962"} Oct 30 21:33:24.937635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:24.937610 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"9a8db61d-594f-4273-b9bf-228e1f011484","Type":"ContainerStarted","Data":"e4d41b53f18a786fe31deab81bb2bbbc3063ca53f7a9cf9574c4367072c21b72"} Oct 30 21:33:25.944217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:25.944179 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" event={"ID":"a1931617-7e55-450f-9c2d-2e058fc3b23e","Type":"ContainerStarted","Data":"9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9"} Oct 30 21:33:25.944713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:25.944492 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:25.944713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:25.944537 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:25.951131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:25.951099 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:25.953705 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:25.953677 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:33:26.165474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:26.165393 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" podStartSLOduration=1.567142545 podStartE2EDuration="12.165378581s" totalImagesPullingTime="10.598236036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:26.114995309 +0000 UTC m=+3434.222491218" watchObservedRunningTime="2026-10-30 21:33:26.165378581 +0000 UTC m=+3434.272874491" Oct 30 21:33:26.407801 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:33:26.407725 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:33:28.964083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:28.964039 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"9a8db61d-594f-4273-b9bf-228e1f011484","Type":"ContainerStarted","Data":"1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74"} Oct 30 21:33:28.964631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:28.964106 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:28.965742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:28.965711 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"4b7513df-1500-4aa5-9f16-d20fc090fbc9","Type":"ContainerStarted","Data":"62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8"} Oct 30 21:33:28.965889 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:28.965770 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:33:29.083056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:29.082998 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podStartSLOduration=12.130634666 podStartE2EDuration="16.082983246s" totalImagesPullingTime="3.95234858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:29.082705018 +0000 UTC m=+3437.190200929" watchObservedRunningTime="2026-10-30 21:33:29.082983246 +0000 UTC m=+3437.190479149" Oct 30 21:33:29.202184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:29.202116 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podStartSLOduration=13.292515687 podStartE2EDuration="17.202098435s" totalImagesPullingTime="3.909582748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:29.201681364 +0000 UTC m=+3437.309177275" watchObservedRunningTime="2026-10-30 21:33:29.202098435 +0000 UTC m=+3437.309594346" Oct 30 21:33:37.431809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:37.431768 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:37.582623 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:37.582574 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:49.975577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:49.975535 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:33:49.976126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:49.976025 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:33:52.471702 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:52.471607 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:33:56.394600 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:33:56.394565 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:33:59.974902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:33:59.974859 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:34:09.974861 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:34:09.974820 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:34:19.976278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:34:19.976238 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:34:20.659838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:34:20.659796 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:21.298341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:34:21.298299 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:34:26.394482 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:34:26.394437 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:34:29.975541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:34:29.975512 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:34:56.394963 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:34:56.394919 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:35:01.731940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:01.731896 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:01.875999 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:01.875952 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:02.001922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.001821 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5"] Oct 30 21:35:02.005271 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.005244 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.028484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.028446 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:02.052987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.052943 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.053185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.052996 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45j6z\" (UniqueName: \"kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.053257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.053207 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.107332 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.107294 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-2lrnh\"" Oct 30 21:35:02.108631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.108605 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5"] Oct 30 21:35:02.111592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.111566 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 21:35:02.111689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.111670 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 21:35:02.153782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.153745 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.154022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.153817 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.154022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.153846 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45j6z\" (UniqueName: \"kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.154539 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.154513 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.156214 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.156194 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.215028 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.214986 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45j6z\" (UniqueName: \"kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z\") pod \"data-science-usage-logs-collector-5b8b69c67b-f2fp5\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.318053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.317954 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:02.691178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.691144 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5"] Oct 30 21:35:02.694391 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:35:02.694350 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d6d621_5044_4285_8a40_c3633e7f400d.slice/crio-7852c7df4fda1ee2b345a121db688d59e3acc04f11d5a504798b8e3ff865c252 WatchSource:0}: Error finding container 7852c7df4fda1ee2b345a121db688d59e3acc04f11d5a504798b8e3ff865c252: Status 404 returned error can't find the container with id 7852c7df4fda1ee2b345a121db688d59e3acc04f11d5a504798b8e3ff865c252 Oct 30 21:35:02.696724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.696681 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:02.696844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:02.696764 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:03.386049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:03.386013 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" event={"ID":"28d6d621-5044-4285-8a40-c3633e7f400d","Type":"ContainerStarted","Data":"1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527"} Oct 30 21:35:03.386474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:03.386064 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" event={"ID":"28d6d621-5044-4285-8a40-c3633e7f400d","Type":"ContainerStarted","Data":"7852c7df4fda1ee2b345a121db688d59e3acc04f11d5a504798b8e3ff865c252"} Oct 30 21:35:03.509645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:03.509591 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" podStartSLOduration=2.509577179 podStartE2EDuration="2.509577179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:03.508232704 +0000 UTC m=+3531.615728613" watchObservedRunningTime="2026-10-30 21:35:03.509577179 +0000 UTC m=+3531.617073087" Oct 30 21:35:08.920036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:08.920003 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:09.196767 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:09.196669 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:11.338102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:11.338067 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:11.481964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:11.481922 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:11.628395 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:11.628303 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:12.250098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.250060 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5"] Oct 30 21:35:12.253611 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.253591 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.334159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.334117 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cds\" (UniqueName: \"kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.334159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.334166 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.334379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.334320 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.356986 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.356951 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5"] Oct 30 21:35:12.434990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.434956 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cds\" (UniqueName: \"kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.434990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.434999 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.435223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.435060 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.435887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.435855 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.437449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.437427 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.496603 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.496562 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cds\" (UniqueName: \"kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds\") pod \"data-science-usage-logs-collector-58dc4d7b85-vqrp5\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.566191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.566098 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:35:12.601027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.600987 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:12.796397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.796359 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:12.946625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.946590 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5"] Oct 30 21:35:12.950187 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:35:12.950152 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ce72bc_bdfd_4232_9a5d_3e7542396b62.slice/crio-511155e564e198b6d6991b10bcb8eb80b6af8f08a993ac4c7a73cd5b868f61e1 WatchSource:0}: Error finding container 511155e564e198b6d6991b10bcb8eb80b6af8f08a993ac4c7a73cd5b868f61e1: Status 404 returned error can't find the container with id 511155e564e198b6d6991b10bcb8eb80b6af8f08a993ac4c7a73cd5b868f61e1 Oct 30 21:35:12.952739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.952701 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:12.952846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.952774 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:12.994879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:12.994847 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.140324 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.140284 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.298604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.298508 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.431485 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.431440 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" event={"ID":"f4ce72bc-bdfd-4232-9a5d-3e7542396b62","Type":"ContainerStarted","Data":"30c65b824e626ca00b6e9230aed1577f490423047c8d90c54f84fb32c5661bbd"} Oct 30 21:35:13.431963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.431507 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" event={"ID":"f4ce72bc-bdfd-4232-9a5d-3e7542396b62","Type":"ContainerStarted","Data":"511155e564e198b6d6991b10bcb8eb80b6af8f08a993ac4c7a73cd5b868f61e1"} Oct 30 21:35:13.447387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.447349 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.548792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.548676 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" podStartSLOduration=1.5486600940000002 podStartE2EDuration="1.548660094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:13.547769024 +0000 UTC m=+3541.655264935" watchObservedRunningTime="2026-10-30 21:35:13.548660094 +0000 UTC m=+3541.656156005" Oct 30 21:35:13.580955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.580908 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5"] Oct 30 21:35:13.581170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.581125 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" podUID="28d6d621-5044-4285-8a40-c3633e7f400d" containerName="otc-container" containerID="cri-o://1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527" gracePeriod=30 Oct 30 21:35:13.601480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.601438 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.748128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.748076 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.849940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.849909 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:13.894060 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.894012 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:13.945148 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.945115 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs\") pod \"28d6d621-5044-4285-8a40-c3633e7f400d\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " Oct 30 21:35:13.945334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.945160 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z\" (UniqueName: \"kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z\") pod \"28d6d621-5044-4285-8a40-c3633e7f400d\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " Oct 30 21:35:13.945334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.945209 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal\") pod \"28d6d621-5044-4285-8a40-c3633e7f400d\" (UID: \"28d6d621-5044-4285-8a40-c3633e7f400d\") " Oct 30 21:35:13.945633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.945601 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal" pod "28d6d621-5044-4285-8a40-c3633e7f400d" (UID: "28d6d621-5044-4285-8a40-c3633e7f400d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:13.947372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.947339 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs" pod "28d6d621-5044-4285-8a40-c3633e7f400d" (UID: "28d6d621-5044-4285-8a40-c3633e7f400d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:13.947372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:13.947354 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z" pod "28d6d621-5044-4285-8a40-c3633e7f400d" (UID: "28d6d621-5044-4285-8a40-c3633e7f400d"). InnerVolumeSpecName "kube-api-access-45j6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:14.045910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.045869 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/28d6d621-5044-4285-8a40-c3633e7f400d-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:14.045910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.045902 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45j6z\" (UniqueName: \"kubernetes.io/projected/28d6d621-5044-4285-8a40-c3633e7f400d-kube-api-access-45j6z\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:14.045910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.045914 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/28d6d621-5044-4285-8a40-c3633e7f400d-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:14.056236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.056202 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:14.169017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.168979 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v"] Oct 30 21:35:14.169328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.169316 4988 cpu_manager.go:414] "containerMap: removing container" podUID="28d6d621-5044-4285-8a40-c3633e7f400d" containerName="otc-container" Oct 30 21:35:14.169388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.169329 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d6d621-5044-4285-8a40-c3633e7f400d" containerName="otc-container" Oct 30 21:35:14.169460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.169419 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d6d621-5044-4285-8a40-c3633e7f400d" containerName="otc-container" Oct 30 21:35:14.172512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.172493 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.228166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.228051 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 21:35:14.228166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.228117 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 21:35:14.228166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.228137 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 21:35:14.228513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.228358 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 21:35:14.228642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.228622 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 21:35:14.228723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.228705 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 21:35:14.234323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.234288 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v"] Oct 30 21:35:14.247612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247575 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.247612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247615 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.247931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247726 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.247931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247760 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.247931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247796 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.247931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247842 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.247931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247899 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.248209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.247988 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwz6c\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.283117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.283066 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:14.348749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348709 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.348749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348751 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348774 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348796 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348823 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348837 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348855 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.348927 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwz6c\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.348973 4988 projected.go:304] Couldn't get configMap payload redhat-ods-monitoring/data-science-lokistack-gateway-ca-bundle: configmap references non-existent config key: service-ca.crt Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.348993 4988 projected.go:196] Error preparing data for projected volume data-science-lokistack-gateway-ca-bundle for pod redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v: configmap references non-existent config key: service-ca.crt Oct 30 21:35:14.349019 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.349010 4988 projected.go:266] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-http: secret "data-science-lokistack-gateway-http" not found Oct 30 21:35:14.349550 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.349038 4988 projected.go:266] Couldn't get secret redhat-ods-monitoring/data-science-lokistack-gateway-http: secret "data-science-lokistack-gateway-http" not found Oct 30 21:35:14.349550 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.349053 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle podName:74403fff-f5c0-440f-8871-1f5b58377390 nodeName:}" failed. No retries permitted until 2026-10-30 21:35:14.849035375 +0000 UTC m=+3542.956531264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-lokistack-gateway-ca-bundle" (UniqueName: "kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle") pod "data-science-lokistack-gateway-8656d54b6-svm2v" (UID: "74403fff-f5c0-440f-8871-1f5b58377390") : configmap references non-existent config key: service-ca.crt Oct 30 21:35:14.349550 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.349051 4988 projected.go:196] Error preparing data for projected volume tls-secret for pod redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v: secret "data-science-lokistack-gateway-http" not found Oct 30 21:35:14.349550 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.349093 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret podName:74403fff-f5c0-440f-8871-1f5b58377390 nodeName:}" failed. No retries permitted until 2026-10-30 21:35:14.84908466 +0000 UTC m=+3542.956580555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-secret" (UniqueName: "kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret") pod "data-science-lokistack-gateway-8656d54b6-svm2v" (UID: "74403fff-f5c0-440f-8871-1f5b58377390") : secret "data-science-lokistack-gateway-http" not found Oct 30 21:35:14.349772 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.349621 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349772 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.349638 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.349772 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.349755 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.351337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.351315 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.351468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.351425 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.409884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.409852 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwz6c\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.437764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.437674 4988 generic.go:356] "Generic (PLEG): container finished" podID="28d6d621-5044-4285-8a40-c3633e7f400d" containerID="1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527" exitCode=0 Oct 30 21:35:14.437764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.437727 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" Oct 30 21:35:14.437764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.437748 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" event={"ID":"28d6d621-5044-4285-8a40-c3633e7f400d","Type":"ContainerDied","Data":"1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527"} Oct 30 21:35:14.438327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.437806 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5" event={"ID":"28d6d621-5044-4285-8a40-c3633e7f400d","Type":"ContainerDied","Data":"7852c7df4fda1ee2b345a121db688d59e3acc04f11d5a504798b8e3ff865c252"} Oct 30 21:35:14.438327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.437828 4988 scope.go:122] "RemoveContainer" containerID="1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527" Oct 30 21:35:14.454397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.454369 4988 scope.go:122] "RemoveContainer" containerID="1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527" Oct 30 21:35:14.454830 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:14.454809 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527\": container with ID starting with 1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527 not found: ID does not exist" containerID="1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527" Oct 30 21:35:14.454910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.454839 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527"} err="failed to get container status \"1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527\": rpc error: code = NotFound desc = could not find container \"1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527\": container with ID starting with 1dc59e1c6c55c00fed332631b297e9653ca1a884c6677765ee5894f2d2e8f527 not found: ID does not exist" Oct 30 21:35:14.500729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.500689 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:14.601701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.601663 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5"] Oct 30 21:35:14.607667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.607632 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b8b69c67b-f2fp5"] Oct 30 21:35:14.851809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.851716 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.851809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.851757 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.854238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.854203 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:14.854351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:14.854278 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\") pod \"data-science-lokistack-gateway-8656d54b6-svm2v\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:15.086132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:15.086093 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:15.270631 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:15.270591 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:15.462390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:15.462293 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:15.573444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:15.569229 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v"] Oct 30 21:35:15.577719 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:35:15.577676 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74403fff_f5c0_440f_8871_1f5b58377390.slice/crio-7143a6c0ff85cc98e09f552ee905c2d39dde3af518fe4f32426ea067bf764628 WatchSource:0}: Error finding container 7143a6c0ff85cc98e09f552ee905c2d39dde3af518fe4f32426ea067bf764628: Status 404 returned error can't find the container with id 7143a6c0ff85cc98e09f552ee905c2d39dde3af518fe4f32426ea067bf764628 Oct 30 21:35:15.611341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:15.611306 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.001274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.001178 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.016324 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.016289 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d6d621-5044-4285-8a40-c3633e7f400d" path="/var/lib/kubelet/pods/28d6d621-5044-4285-8a40-c3633e7f400d/volumes" Oct 30 21:35:16.176374 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.176339 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.450108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.450074 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" event={"ID":"74403fff-f5c0-440f-8871-1f5b58377390","Type":"ContainerStarted","Data":"e6a7453aabb41cbeb118adf7f84bdc76567ea6407b71565e2249644ccc2e3f8e"} Oct 30 21:35:16.450310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.450116 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" event={"ID":"74403fff-f5c0-440f-8871-1f5b58377390","Type":"ContainerStarted","Data":"80f28fa83d3400005afbcd71643aae44f38ad0580017da624ce3ac289300d279"} Oct 30 21:35:16.450310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.450129 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" event={"ID":"74403fff-f5c0-440f-8871-1f5b58377390","Type":"ContainerStarted","Data":"7143a6c0ff85cc98e09f552ee905c2d39dde3af518fe4f32426ea067bf764628"} Oct 30 21:35:16.450310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.450178 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:16.450310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.450201 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:16.456562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.456535 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:16.460175 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.460152 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:35:16.571552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.571496 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" podStartSLOduration=2.571479337 podStartE2EDuration="2.571479337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:16.569792626 +0000 UTC m=+3544.677288549" watchObservedRunningTime="2026-10-30 21:35:16.571479337 +0000 UTC m=+3544.678975247" Oct 30 21:35:16.730270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.730182 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 21:35:16.733800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.733771 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:16.786704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.786640 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\"" Oct 30 21:35:16.786941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.786785 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\"" Oct 30 21:35:16.786941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.786805 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 21:35:16.786941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.786838 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-4gxlb\"" Oct 30 21:35:16.786941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.786858 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-logs-loki-secret\"" Oct 30 21:35:16.794535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.794023 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 21:35:16.896674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.896630 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:16.968666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.968630 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:16.968829 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.968681 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:16.968829 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.968701 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:16.968919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.968834 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:16.968919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.968864 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:16.968986 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:16.968943 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.069986 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070035 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070052 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070081 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070448 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070100 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070448 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070131 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.070940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070908 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.071057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.070979 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.072634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.072608 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.072745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.072670 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.073255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.073234 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.132460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.132404 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:17.302644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.302609 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-vdlvj"] Oct 30 21:35:17.307164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.307136 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.367923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.367829 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-vdlvj"] Oct 30 21:35:17.388751 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.388714 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 30 21:35:17.388933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.388881 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 30 21:35:17.388933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.388906 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-hh4ns\"" Oct 30 21:35:17.441648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.441615 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 30 21:35:17.441976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.441958 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 30 21:35:17.467280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.467240 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:17.473231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473201 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473233 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4vf\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473261 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473318 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473420 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473477 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473511 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473531 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473555 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473592 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.473835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.473617 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.574859 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.574820 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.574866 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4vf\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.574931 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.574962 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.574991 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575065 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575093 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575141 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575144 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575191 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:17.575223 4988 secret.go:189] Couldn't get secret openshift-logging/data-science-cluster-log-forwarder-metrics: secret "data-science-cluster-log-forwarder-metrics" not found Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575226 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575257 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.575353 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:17.575292 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics podName:d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577 nodeName:}" failed. No retries permitted until 2026-10-30 21:35:18.075270726 +0000 UTC m=+3546.182766631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics" (UniqueName: "kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics") pod "data-science-cluster-log-forwarder-vdlvj" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577") : secret "data-science-cluster-log-forwarder-metrics" not found Oct 30 21:35:17.576362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575434 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.576362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.575991 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.576362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.576053 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.576362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.576283 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.576607 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.576586 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.577567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.577547 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.577830 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.577812 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.640594 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.640559 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4vf\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.640885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.640865 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:17.840238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:17.840199 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:18.081108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:18.081009 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:18.083651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:18.083622 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\") pod \"data-science-cluster-log-forwarder-vdlvj\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:18.220622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:18.220579 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:18.231459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:18.231403 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-vdlvj"] Oct 30 21:35:18.660128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:18.660093 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-vdlvj"] Oct 30 21:35:18.695327 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:35:18.695283 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ec6fcf_6d17_4a2d_b8d8_ae0b64be2577.slice/crio-28f3fecc147ef8fe246e629d312c7f7ac7c5f396d294b3a3c7523f475e949100 WatchSource:0}: Error finding container 28f3fecc147ef8fe246e629d312c7f7ac7c5f396d294b3a3c7523f475e949100: Status 404 returned error can't find the container with id 28f3fecc147ef8fe246e629d312c7f7ac7c5f396d294b3a3c7523f475e949100 Oct 30 21:35:19.465369 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:19.465329 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" event={"ID":"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577","Type":"ContainerStarted","Data":"28f3fecc147ef8fe246e629d312c7f7ac7c5f396d294b3a3c7523f475e949100"} Oct 30 21:35:21.660494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:21.660454 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:35:25.498180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.498132 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" event={"ID":"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577","Type":"ContainerStarted","Data":"8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d"} Oct 30 21:35:25.498180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.498167 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" podUID="d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" containerName="collector" containerID="cri-o://8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d" gracePeriod=10 Oct 30 21:35:25.648026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.647987 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:25.667946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.667882 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" podStartSLOduration=2.381189257 podStartE2EDuration="8.667866693s" totalImagesPullingTime="6.286677436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:25.667005147 +0000 UTC m=+3553.774501057" watchObservedRunningTime="2026-10-30 21:35:25.667866693 +0000 UTC m=+3553.775362604" Oct 30 21:35:25.752871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.752771 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:25.782885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.782852 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:25.850761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850724 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850785 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850815 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850841 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850872 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850842 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850921 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.850944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.850866 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 21:35:25.851316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851009 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.851316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851070 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.851316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851113 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.851316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851142 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.851316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851195 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token\") pod \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\" (UID: \"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577\") " Oct 30 21:35:25.851316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851302 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:25.851668 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851391 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:25.851668 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851445 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:25.851668 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851537 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:25.851668 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851648 4988 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-varlogpods\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.851897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851670 4988 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-datadir\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.851897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851700 4988 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config-openshift-service-cacrt\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.851897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851716 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.851897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.851762 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:25.853252 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.853223 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:25.853591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.853556 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:25.853717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.853595 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:25.853717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.853699 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:25.853852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.853835 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "kube-api-access-6g4vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:25.856743 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.856721 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint" pod "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" (UID: "d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:25.903905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.903863 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") DevicePath \"csi-9a59dccc0527d335dc762f5246b5bff497cc704674444741ea92776472726335\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:25.952210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952177 4988 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-tmp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.952210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952208 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4vf\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-kube-api-access-6g4vf\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.952210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952218 4988 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-entrypoint\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.952458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952228 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.952458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952241 4988 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-sa-token\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.952458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952250 4988 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-metrics\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.952458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.952259 4988 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577-trusted-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:25.987731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.987692 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-8c2n4"] Oct 30 21:35:25.988069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.988057 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" containerName="collector" Oct 30 21:35:25.988117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.988070 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" containerName="collector" Oct 30 21:35:25.988150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.988128 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" containerName="collector" Oct 30 21:35:25.992266 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:25.992240 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.053607 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.053552 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.053821 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.053708 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.053985 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054042 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054152 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054436 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054481 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054517 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054572 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9jj\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054720 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.054901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.054858 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.055836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.055813 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-8c2n4"] Oct 30 21:35:26.156056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156015 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156115 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156150 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156168 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156199 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156231 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156233 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156629 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156260 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156629 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156312 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156629 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156332 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156629 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156353 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156629 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156377 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9jj\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.156940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156744 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.157011 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.156985 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.157073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.157010 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.157333 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.157313 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.157523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.157506 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.158854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.158832 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.158991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.158947 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.159730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.159709 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.160009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.159973 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1a96f0603d54ab5a31fd9a6b776a4563f87aa1e1c72eb9f0a206a9a426698aa9/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:26.218814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.218780 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.219191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.219172 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9jj\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj\") pod \"data-science-cluster-log-forwarder-8c2n4\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.223101 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.223080 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:26.309787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.309692 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:35:26.344998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.344960 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:26.420962 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:26.420689 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:35:26.504096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.504053 4988 generic.go:356] "Generic (PLEG): container finished" podID="d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" containerID="8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d" exitCode=0 Oct 30 21:35:26.504565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.504104 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" Oct 30 21:35:26.504565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.504138 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" event={"ID":"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577","Type":"ContainerDied","Data":"8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d"} Oct 30 21:35:26.504565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.504194 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-vdlvj" event={"ID":"d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577","Type":"ContainerDied","Data":"28f3fecc147ef8fe246e629d312c7f7ac7c5f396d294b3a3c7523f475e949100"} Oct 30 21:35:26.504565 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.504217 4988 scope.go:122] "RemoveContainer" containerID="8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d" Oct 30 21:35:26.513305 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.513281 4988 scope.go:122] "RemoveContainer" containerID="8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d" Oct 30 21:35:26.513671 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:26.513651 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d\": container with ID starting with 8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d not found: ID does not exist" containerID="8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d" Oct 30 21:35:26.513740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.513681 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d"} err="failed to get container status \"8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d\": rpc error: code = NotFound desc = could not find container \"8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d\": container with ID starting with 8d37588ffb8875d80318c330194259014e8c8ff433f9cbceb70ba1b0e4cb205d not found: ID does not exist" Oct 30 21:35:26.678326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.678291 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-vdlvj"] Oct 30 21:35:26.727494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.727459 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-vdlvj"] Oct 30 21:35:26.743257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.743231 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-8c2n4"] Oct 30 21:35:26.745847 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:35:26.745817 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bb886e_4838_4858_86c6_6ea7028e5978.slice/crio-3982c100f14ae3f544c8d5e5c69c434678b5493fc1b1d2c9a00d488eec1457aa WatchSource:0}: Error finding container 3982c100f14ae3f544c8d5e5c69c434678b5493fc1b1d2c9a00d488eec1457aa: Status 404 returned error can't find the container with id 3982c100f14ae3f544c8d5e5c69c434678b5493fc1b1d2c9a00d488eec1457aa Oct 30 21:35:26.752510 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:26.752474 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 21:35:26.781585 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:35:26.781549 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8917129_9882_4549_ad37_10938b5a3938.slice/crio-a430a12d26d1c116300bb6309279944b1c06b95481cefe6cc98938642b8ebcc3 WatchSource:0}: Error finding container a430a12d26d1c116300bb6309279944b1c06b95481cefe6cc98938642b8ebcc3: Status 404 returned error can't find the container with id a430a12d26d1c116300bb6309279944b1c06b95481cefe6cc98938642b8ebcc3 Oct 30 21:35:27.511082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.511046 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f8917129-9882-4549-ad37-10938b5a3938","Type":"ContainerStarted","Data":"7f23356e593f25ab4970ff4026f0d20b1d326743227caca682131bad78576b05"} Oct 30 21:35:27.511717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.511092 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f8917129-9882-4549-ad37-10938b5a3938","Type":"ContainerStarted","Data":"a430a12d26d1c116300bb6309279944b1c06b95481cefe6cc98938642b8ebcc3"} Oct 30 21:35:27.511717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.511256 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:27.512773 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.512746 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" event={"ID":"b0bb886e-4838-4858-86c6-6ea7028e5978","Type":"ContainerStarted","Data":"c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8"} Oct 30 21:35:27.512886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.512795 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" event={"ID":"b0bb886e-4838-4858-86c6-6ea7028e5978","Type":"ContainerStarted","Data":"3982c100f14ae3f544c8d5e5c69c434678b5493fc1b1d2c9a00d488eec1457aa"} Oct 30 21:35:27.631678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.631619 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=14.631602849 podStartE2EDuration="14.631602849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:27.630214163 +0000 UTC m=+3555.737710074" watchObservedRunningTime="2026-10-30 21:35:27.631602849 +0000 UTC m=+3555.739098760" Oct 30 21:35:27.751795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:27.751740 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" podStartSLOduration=2.751723255 podStartE2EDuration="2.751723255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:27.750157843 +0000 UTC m=+3555.857653769" watchObservedRunningTime="2026-10-30 21:35:27.751723255 +0000 UTC m=+3555.859219145" Oct 30 21:35:28.016032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:28.015988 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577" path="/var/lib/kubelet/pods/d7ec6fcf-6d17-4a2d-b8d8-ae0b64be2577/volumes" Oct 30 21:35:48.522844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:35:48.522812 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:35:56.395347 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:35:56.395310 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:36:12.366614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:12.366496 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:36:12.373319 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:12.373289 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:36:21.996299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:21.996252 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:26.394961 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:36:26.394919 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:36:48.258387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.258350 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.417910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.417875 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.615275 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.615178 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.694010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.693972 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 21:36:48.694198 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.694161 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="f8917129-9882-4549-ad37-10938b5a3938" containerName="loki-index-gateway" containerID="cri-o://7f23356e593f25ab4970ff4026f0d20b1d326743227caca682131bad78576b05" gracePeriod=30 Oct 30 21:36:48.763729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.763695 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v"] Oct 30 21:36:48.763917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.763895 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="gateway" containerID="cri-o://80f28fa83d3400005afbcd71643aae44f38ad0580017da624ce3ac289300d279" gracePeriod=30 Oct 30 21:36:48.763963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.763924 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="opa" containerID="cri-o://e6a7453aabb41cbeb118adf7f84bdc76567ea6407b71565e2249644ccc2e3f8e" gracePeriod=30 Oct 30 21:36:48.787500 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.787464 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:48.879335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.879292 4988 generic.go:356] "Generic (PLEG): container finished" podID="f8917129-9882-4549-ad37-10938b5a3938" containerID="7f23356e593f25ab4970ff4026f0d20b1d326743227caca682131bad78576b05" exitCode=0 Oct 30 21:36:48.879526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:48.879335 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f8917129-9882-4549-ad37-10938b5a3938","Type":"ContainerDied","Data":"7f23356e593f25ab4970ff4026f0d20b1d326743227caca682131bad78576b05"} Oct 30 21:36:49.015739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.015710 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:36:49.116855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.116817 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.116855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.116860 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.117108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.116964 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.117108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.116993 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.117108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.117035 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.117108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.117066 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.117108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.117104 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc\") pod \"f8917129-9882-4549-ad37-10938b5a3938\" (UID: \"f8917129-9882-4549-ad37-10938b5a3938\") " Oct 30 21:36:49.117468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.117374 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:49.117627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.117572 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:49.119294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.119261 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:49.119394 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.119348 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:49.119589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.119564 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "kube-api-access-9bgs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:49.119689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.119651 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "test-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:49.119925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.119909 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c" pod "f8917129-9882-4549-ad37-10938b5a3938" (UID: "f8917129-9882-4549-ad37-10938b5a3938"). InnerVolumeSpecName "pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:36:49.217848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217810 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:49.217848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217847 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:49.217848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217858 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:49.218105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217868 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgs2\" (UniqueName: \"kubernetes.io/projected/f8917129-9882-4549-ad37-10938b5a3938-kube-api-access-9bgs2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:49.218105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217899 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:36:49.218105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217909 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:49.218105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.217923 4988 reconciler_common.go:299] "Volume detached for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f8917129-9882-4549-ad37-10938b5a3938-test-logs-loki-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:49.231168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.231145 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:36:49.319014 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.318967 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-c5c4420b-8879-47ab-88b8-5a9285bc41b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-034db0abb5c7a3a6c\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-9a59dccc0527d335dc762f5246b5bff497cc704674444741ea92776472726335\"" Oct 30 21:36:49.449228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.449185 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-8c2n4"] Oct 30 21:36:49.449466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.449426 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" podUID="b0bb886e-4838-4858-86c6-6ea7028e5978" containerName="collector" containerID="cri-o://c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8" gracePeriod=10 Oct 30 21:36:49.481007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.480967 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:49.885865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.885839 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" Oct 30 21:36:49.886020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.885858 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f8917129-9882-4549-ad37-10938b5a3938","Type":"ContainerDied","Data":"a430a12d26d1c116300bb6309279944b1c06b95481cefe6cc98938642b8ebcc3"} Oct 30 21:36:49.886020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.885921 4988 scope.go:122] "RemoveContainer" containerID="7f23356e593f25ab4970ff4026f0d20b1d326743227caca682131bad78576b05" Oct 30 21:36:49.889097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.889066 4988 generic.go:356] "Generic (PLEG): container finished" podID="74403fff-f5c0-440f-8871-1f5b58377390" containerID="e6a7453aabb41cbeb118adf7f84bdc76567ea6407b71565e2249644ccc2e3f8e" exitCode=0 Oct 30 21:36:49.889211 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.889101 4988 generic.go:356] "Generic (PLEG): container finished" podID="74403fff-f5c0-440f-8871-1f5b58377390" containerID="80f28fa83d3400005afbcd71643aae44f38ad0580017da624ce3ac289300d279" exitCode=0 Oct 30 21:36:49.889211 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.889142 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" event={"ID":"74403fff-f5c0-440f-8871-1f5b58377390","Type":"ContainerDied","Data":"e6a7453aabb41cbeb118adf7f84bdc76567ea6407b71565e2249644ccc2e3f8e"} Oct 30 21:36:49.889211 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:49.889190 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" event={"ID":"74403fff-f5c0-440f-8871-1f5b58377390","Type":"ContainerDied","Data":"80f28fa83d3400005afbcd71643aae44f38ad0580017da624ce3ac289300d279"} Oct 30 21:36:50.034267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.034238 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:36:50.074387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.074348 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 21:36:50.090246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.090212 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"] Oct 30 21:36:50.125502 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125467 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125517 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125556 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125579 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125636 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125671 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125708 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.125922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125758 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants\") pod \"74403fff-f5c0-440f-8871-1f5b58377390\" (UID: \"74403fff-f5c0-440f-8871-1f5b58377390\") " Oct 30 21:36:50.126039 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.125969 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:50.126116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.126097 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.128282 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.128250 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "kube-api-access-mwz6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:50.128466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.128389 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:50.128585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.128462 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:50.129255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.128712 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:50.132955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.132853 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:50.133095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.133031 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:50.133974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.133948 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants" pod "74403fff-f5c0-440f-8871-1f5b58377390" (UID: "74403fff-f5c0-440f-8871-1f5b58377390"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:50.227367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227328 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.227367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227365 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwz6c\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-kube-api-access-mwz6c\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.227367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227376 4988 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-lokistack-gateway\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.227626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227386 4988 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74403fff-f5c0-440f-8871-1f5b58377390-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.227626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227395 4988 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74403fff-f5c0-440f-8871-1f5b58377390-tls-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.227626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227405 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.227626 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.227443 4988 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74403fff-f5c0-440f-8871-1f5b58377390-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:50.433226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.433193 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:36:50.896354 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.896323 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" Oct 30 21:36:50.896573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.896357 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v" event={"ID":"74403fff-f5c0-440f-8871-1f5b58377390","Type":"ContainerDied","Data":"7143a6c0ff85cc98e09f552ee905c2d39dde3af518fe4f32426ea067bf764628"} Oct 30 21:36:50.896573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.896428 4988 scope.go:122] "RemoveContainer" containerID="e6a7453aabb41cbeb118adf7f84bdc76567ea6407b71565e2249644ccc2e3f8e" Oct 30 21:36:50.904858 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:50.904841 4988 scope.go:122] "RemoveContainer" containerID="80f28fa83d3400005afbcd71643aae44f38ad0580017da624ce3ac289300d279" Oct 30 21:36:51.062482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:51.062391 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v"] Oct 30 21:36:51.068115 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:51.068090 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-8656d54b6-svm2v"] Oct 30 21:36:51.690733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:51.690691 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5"] Oct 30 21:36:51.691214 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:51.690920 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" podUID="f4ce72bc-bdfd-4232-9a5d-3e7542396b62" containerName="otc-container" containerID="cri-o://30c65b824e626ca00b6e9230aed1577f490423047c8d90c54f84fb32c5661bbd" gracePeriod=30 Oct 30 21:36:51.903808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:51.903689 4988 generic.go:356] "Generic (PLEG): container finished" podID="f4ce72bc-bdfd-4232-9a5d-3e7542396b62" containerID="30c65b824e626ca00b6e9230aed1577f490423047c8d90c54f84fb32c5661bbd" exitCode=0 Oct 30 21:36:51.903808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:51.903773 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" event={"ID":"f4ce72bc-bdfd-4232-9a5d-3e7542396b62","Type":"ContainerDied","Data":"30c65b824e626ca00b6e9230aed1577f490423047c8d90c54f84fb32c5661bbd"} Oct 30 21:36:52.017333 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.017260 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74403fff-f5c0-440f-8871-1f5b58377390" path="/var/lib/kubelet/pods/74403fff-f5c0-440f-8871-1f5b58377390/volumes" Oct 30 21:36:52.017744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.017728 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8917129-9882-4549-ad37-10938b5a3938" path="/var/lib/kubelet/pods/f8917129-9882-4549-ad37-10938b5a3938/volumes" Oct 30 21:36:52.050841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.050816 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:36:52.145368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.145332 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs\") pod \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " Oct 30 21:36:52.145563 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.145390 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds\" (UniqueName: \"kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds\") pod \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " Oct 30 21:36:52.145563 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.145447 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal\") pod \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\" (UID: \"f4ce72bc-bdfd-4232-9a5d-3e7542396b62\") " Oct 30 21:36:52.145863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.145830 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal" pod "f4ce72bc-bdfd-4232-9a5d-3e7542396b62" (UID: "f4ce72bc-bdfd-4232-9a5d-3e7542396b62"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:52.147562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.147543 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs" pod "f4ce72bc-bdfd-4232-9a5d-3e7542396b62" (UID: "f4ce72bc-bdfd-4232-9a5d-3e7542396b62"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:52.147648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.147614 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds" pod "f4ce72bc-bdfd-4232-9a5d-3e7542396b62" (UID: "f4ce72bc-bdfd-4232-9a5d-3e7542396b62"). InnerVolumeSpecName "kube-api-access-88cds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:52.246437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.246366 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88cds\" (UniqueName: \"kubernetes.io/projected/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-kube-api-access-88cds\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:52.246437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.246400 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:52.246437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.246438 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ce72bc-bdfd-4232-9a5d-3e7542396b62-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:52.910884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.910852 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" Oct 30 21:36:52.911340 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.910858 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5" event={"ID":"f4ce72bc-bdfd-4232-9a5d-3e7542396b62","Type":"ContainerDied","Data":"511155e564e198b6d6991b10bcb8eb80b6af8f08a993ac4c7a73cd5b868f61e1"} Oct 30 21:36:52.911340 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:52.910989 4988 scope.go:122] "RemoveContainer" containerID="30c65b824e626ca00b6e9230aed1577f490423047c8d90c54f84fb32c5661bbd" Oct 30 21:36:53.079154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:53.079119 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5"] Oct 30 21:36:53.127823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:53.127789 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-58dc4d7b85-vqrp5"] Oct 30 21:36:54.015642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:54.015605 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ce72bc-bdfd-4232-9a5d-3e7542396b62" path="/var/lib/kubelet/pods/f4ce72bc-bdfd-4232-9a5d-3e7542396b62/volumes" Oct 30 21:36:56.393325 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:36:56.393286 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:36:59.257653 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.257604 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 21:36:59.258162 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.257802 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" containerID="cri-o://62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8" gracePeriod=600 Oct 30 21:36:59.431291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.431257 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:36:59.431493 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.431473 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="9a8db61d-594f-4273-b9bf-228e1f011484" containerName="loki-compactor" containerID="cri-o://1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74" gracePeriod=30 Oct 30 21:36:59.432627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.432601 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5"] Oct 30 21:36:59.432974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.432958 4988 cpu_manager.go:414] "containerMap: removing container" podUID="f8917129-9882-4549-ad37-10938b5a3938" containerName="loki-index-gateway" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.432976 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8917129-9882-4549-ad37-10938b5a3938" containerName="loki-index-gateway" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.432992 4988 cpu_manager.go:414] "containerMap: removing container" podUID="f4ce72bc-bdfd-4232-9a5d-3e7542396b62" containerName="otc-container" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433000 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ce72bc-bdfd-4232-9a5d-3e7542396b62" containerName="otc-container" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433018 4988 cpu_manager.go:414] "containerMap: removing container" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="gateway" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433025 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="gateway" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433038 4988 cpu_manager.go:414] "containerMap: removing container" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="opa" Oct 30 21:36:59.433053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433045 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="opa" Oct 30 21:36:59.433388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433134 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8917129-9882-4549-ad37-10938b5a3938" containerName="loki-index-gateway" Oct 30 21:36:59.433388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433148 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ce72bc-bdfd-4232-9a5d-3e7542396b62" containerName="otc-container" Oct 30 21:36:59.433388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433159 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="gateway" Oct 30 21:36:59.433388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.433168 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74403fff-f5c0-440f-8871-1f5b58377390" containerName="opa" Oct 30 21:36:59.437550 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.437530 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.491456 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.491405 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 21:36:59.493939 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.493913 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 30 21:36:59.496902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.496874 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5"] Oct 30 21:36:59.513364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.513331 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.513502 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.513378 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.513579 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.513517 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc79f\" (UniqueName: \"kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.513579 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.513556 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.513689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.513585 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.513689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.513651 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.591813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.591783 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 30 21:36:59.614724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.614687 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.614926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.614741 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.614926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.614789 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.614926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.614880 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc79f\" (UniqueName: \"kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.615090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.614923 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.615090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.614966 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.615600 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.615559 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.615730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.615701 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.617533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.617505 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.617688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.617673 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.617781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.617757 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.649757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.649728 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-logging_data-science-cluster-log-forwarder-8c2n4_b0bb886e-4838-4858-86c6-6ea7028e5978/collector/0.log" Oct 30 21:36:59.653100 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.653079 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:36:59.679263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.679224 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc79f\" (UniqueName: \"kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.715908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.715873 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.715908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.715917 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.715972 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.715990 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716014 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716041 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716079 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 21:36:59.716146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716106 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716192 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716244 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716268 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716297 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:59.716513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716301 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt\") pod \"b0bb886e-4838-4858-86c6-6ea7028e5978\" (UID: \"b0bb886e-4838-4858-86c6-6ea7028e5978\") " Oct 30 21:36:59.716513 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716458 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 21:36:59.716839 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716647 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:59.716975 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.716915 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.717144 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.717026 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:59.717267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.717241 4988 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-datadir\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.717379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.717296 4988 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b0bb886e-4838-4858-86c6-6ea7028e5978-varlogpods\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.717379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.717316 4988 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-config-openshift-service-cacrt\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.718470 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.718443 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:59.718598 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.718570 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:59.718700 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.718680 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:59.718765 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.718680 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "kube-api-access-5z9jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:59.719041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.719024 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:59.721712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.721691 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint" pod "b0bb886e-4838-4858-86c6-6ea7028e5978" (UID: "b0bb886e-4838-4858-86c6-6ea7028e5978"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:59.748941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.748903 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:36:59.818139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818101 4988 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-entrypoint\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.818139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818143 4988 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb886e-4838-4858-86c6-6ea7028e5978-trusted-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.818321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818159 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z9jj\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-kube-api-access-5z9jj\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.818321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818176 4988 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b0bb886e-4838-4858-86c6-6ea7028e5978-sa-token\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.818321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818192 4988 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-metrics\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.818321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818205 4988 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0bb886e-4838-4858-86c6-6ea7028e5978-tmp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.818321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.818218 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b0bb886e-4838-4858-86c6-6ea7028e5978-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:59.944776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.944748 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-logging_data-science-cluster-log-forwarder-8c2n4_b0bb886e-4838-4858-86c6-6ea7028e5978/collector/0.log" Oct 30 21:36:59.944973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.944790 4988 generic.go:356] "Generic (PLEG): container finished" podID="b0bb886e-4838-4858-86c6-6ea7028e5978" containerID="c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8" exitCode=137 Oct 30 21:36:59.944973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.944868 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" Oct 30 21:36:59.944973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.944892 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" event={"ID":"b0bb886e-4838-4858-86c6-6ea7028e5978","Type":"ContainerDied","Data":"c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8"} Oct 30 21:36:59.944973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.944949 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-8c2n4" event={"ID":"b0bb886e-4838-4858-86c6-6ea7028e5978","Type":"ContainerDied","Data":"3982c100f14ae3f544c8d5e5c69c434678b5493fc1b1d2c9a00d488eec1457aa"} Oct 30 21:36:59.944973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.944974 4988 scope.go:122] "RemoveContainer" containerID="c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8" Oct 30 21:36:59.954004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.953981 4988 scope.go:122] "RemoveContainer" containerID="c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8" Oct 30 21:36:59.954332 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:36:59.954309 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8\": container with ID starting with c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8 not found: ID does not exist" containerID="c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8" Oct 30 21:36:59.954453 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.954344 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8"} err="failed to get container status \"c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8\": rpc error: code = NotFound desc = could not find container \"c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8\": container with ID starting with c86093a802025299fef052b2377e75f48d767fcb24b0c50e7e58395088a479e8 not found: ID does not exist" Oct 30 21:36:59.975680 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.975637 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:36:59.975862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:36:59.975763 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="9a8db61d-594f-4273-b9bf-228e1f011484" containerName="loki-compactor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:37:00.067288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.067195 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.111191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.111156 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-8c2n4"] Oct 30 21:37:00.118775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.118745 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-8c2n4"] Oct 30 21:37:00.125878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.125851 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5"] Oct 30 21:37:00.129097 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:37:00.129071 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15a8d404_235a_4426_af60_1183867f77ad.slice/crio-474506cb9be6fa1cc028dbf6198443def268c0a15fbd2589f7f2d1daeea9aca8 WatchSource:0}: Error finding container 474506cb9be6fa1cc028dbf6198443def268c0a15fbd2589f7f2d1daeea9aca8: Status 404 returned error can't find the container with id 474506cb9be6fa1cc028dbf6198443def268c0a15fbd2589f7f2d1daeea9aca8 Oct 30 21:37:00.217203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.217172 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:00.307050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.307020 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:37:00.378116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.378084 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:00.423482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423439 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423500 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423625 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423667 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423857 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423731 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423857 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423824 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423864 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.423958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423906 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:00.423958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.423936 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw\" (UniqueName: \"kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw\") pod \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\" (UID: \"4b7513df-1500-4aa5-9f16-d20fc090fbc9\") " Oct 30 21:37:00.424328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.424303 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.424328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.424301 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:00.426331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.426297 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:00.426735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.426658 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:00.426735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.426701 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:00.426919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.426809 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "kube-api-access-mcxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:00.426919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.426854 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:37:00.426919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.426860 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9" pod "4b7513df-1500-4aa5-9f16-d20fc090fbc9" (UID: "4b7513df-1500-4aa5-9f16-d20fc090fbc9"). InnerVolumeSpecName "pvc-095ababd-be79-496f-b4fb-2d81ff245a1a". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:37:00.524764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.524727 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.524952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.524781 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.524952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.524835 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.524952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.524881 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.525145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.524994 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.525145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525036 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.525145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525083 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd\" (UniqueName: \"kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd\") pod \"9a8db61d-594f-4273-b9bf-228e1f011484\" (UID: \"9a8db61d-594f-4273-b9bf-228e1f011484\") " Oct 30 21:37:00.525304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525245 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525527 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525550 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525594 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525613 4988 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-test-usage-logs-loki-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525628 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ingester-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525650 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525665 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b7513df-1500-4aa5-9f16-d20fc090fbc9-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525680 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.527132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.525696 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcxhw\" (UniqueName: \"kubernetes.io/projected/4b7513df-1500-4aa5-9f16-d20fc090fbc9-kube-api-access-mcxhw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.527794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.527732 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:00.527863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.527819 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:00.527863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.527830 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:00.528183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.528162 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:37:00.528273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.528229 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd" pod "9a8db61d-594f-4273-b9bf-228e1f011484" (UID: "9a8db61d-594f-4273-b9bf-228e1f011484"). InnerVolumeSpecName "kube-api-access-vlfmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:00.537552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.537528 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:37:00.540587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.540571 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-095ababd-be79-496f-b4fb-2d81ff245a1a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:37:00.626339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626256 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-095ababd-be79-496f-b4fb-2d81ff245a1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fdad28dd10561ce9\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-df2bbbe12036b89ff6a72b070aa5ff076d9a9f83f6e13ab3c18c3d8609fd3461\"" Oct 30 21:37:00.626339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626287 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.626339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626298 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.626339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626309 4988 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/9a8db61d-594f-4273-b9bf-228e1f011484-test-usage-logs-loki-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.626339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626338 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:37:00.626638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626350 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-b7f34ad9-8b36-4351-b6e4-5eb5bbd79d36\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0d49f1453ab2e1b57\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-2c615d72aa7032fc0973683fa68549afa65f559f301223e7d8c7fbe6ace7eca0\"" Oct 30 21:37:00.626638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626359 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a8db61d-594f-4273-b9bf-228e1f011484-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.626638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.626370 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfmd\" (UniqueName: \"kubernetes.io/projected/9a8db61d-594f-4273-b9bf-228e1f011484-kube-api-access-vlfmd\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:00.634272 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.634251 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:37:00.727736 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.727694 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-155d7e76a00fba862a255a008b4646e5e30bae03a1ce3c419eed77bd55a26b19\"" Oct 30 21:37:00.951059 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.951022 4988 generic.go:356] "Generic (PLEG): container finished" podID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerID="62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8" exitCode=0 Oct 30 21:37:00.951293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.951063 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 30 21:37:00.951293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.951108 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"4b7513df-1500-4aa5-9f16-d20fc090fbc9","Type":"ContainerDied","Data":"62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8"} Oct 30 21:37:00.951293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.951156 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"4b7513df-1500-4aa5-9f16-d20fc090fbc9","Type":"ContainerDied","Data":"29a4199b54cea3d4ed33f0c0c80acfdea6e88446d56ed0a42184f6be1bab7962"} Oct 30 21:37:00.951293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.951181 4988 scope.go:122] "RemoveContainer" containerID="62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8" Oct 30 21:37:00.952854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.952824 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" event={"ID":"15a8d404-235a-4426-af60-1183867f77ad","Type":"ContainerStarted","Data":"cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847"} Oct 30 21:37:00.952977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.952867 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" event={"ID":"15a8d404-235a-4426-af60-1183867f77ad","Type":"ContainerStarted","Data":"474506cb9be6fa1cc028dbf6198443def268c0a15fbd2589f7f2d1daeea9aca8"} Oct 30 21:37:00.952977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.952925 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:37:00.954323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.954301 4988 generic.go:356] "Generic (PLEG): container finished" podID="9a8db61d-594f-4273-b9bf-228e1f011484" containerID="1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74" exitCode=0 Oct 30 21:37:00.954523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.954340 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:00.954523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.954361 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"9a8db61d-594f-4273-b9bf-228e1f011484","Type":"ContainerDied","Data":"1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74"} Oct 30 21:37:00.954523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.954395 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"9a8db61d-594f-4273-b9bf-228e1f011484","Type":"ContainerDied","Data":"e4d41b53f18a786fe31deab81bb2bbbc3063ca53f7a9cf9574c4367072c21b72"} Oct 30 21:37:00.961919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.961895 4988 scope.go:122] "RemoveContainer" containerID="62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8" Oct 30 21:37:00.962251 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:00.962222 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8\": container with ID starting with 62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8 not found: ID does not exist" containerID="62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8" Oct 30 21:37:00.962344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.962261 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8"} err="failed to get container status \"62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8\": rpc error: code = NotFound desc = could not find container \"62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8\": container with ID starting with 62b80d67c904ea102ddc7f8f398df35553801b4f2f0e09432ed1652caad8b3c8 not found: ID does not exist" Oct 30 21:37:00.962344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.962284 4988 scope.go:122] "RemoveContainer" containerID="1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74" Oct 30 21:37:00.970805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.970786 4988 scope.go:122] "RemoveContainer" containerID="1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74" Oct 30 21:37:00.971068 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:00.971051 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74\": container with ID starting with 1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74 not found: ID does not exist" containerID="1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74" Oct 30 21:37:00.971128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:00.971075 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74"} err="failed to get container status \"1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74\": rpc error: code = NotFound desc = could not find container \"1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74\": container with ID starting with 1958d4a8c1dd853d47d4ca398828f9b4ccc77ab97babf0bd8064a6dbf80d5a74 not found: ID does not exist" Oct 30 21:37:01.086773 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.086741 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:01.087086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087074 4988 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb886e-4838-4858-86c6-6ea7028e5978" containerName="collector" Oct 30 21:37:01.087139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087087 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb886e-4838-4858-86c6-6ea7028e5978" containerName="collector" Oct 30 21:37:01.087139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087102 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" Oct 30 21:37:01.087139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087108 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" Oct 30 21:37:01.087139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087130 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9a8db61d-594f-4273-b9bf-228e1f011484" containerName="loki-compactor" Oct 30 21:37:01.087139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087135 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8db61d-594f-4273-b9bf-228e1f011484" containerName="loki-compactor" Oct 30 21:37:01.087296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087201 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8db61d-594f-4273-b9bf-228e1f011484" containerName="loki-compactor" Oct 30 21:37:01.087296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087211 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb886e-4838-4858-86c6-6ea7028e5978" containerName="collector" Oct 30 21:37:01.087296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.087218 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" containerName="loki-ingester" Oct 30 21:37:01.091797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.091773 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.130462 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.125307 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" podStartSLOduration=2.125287065 podStartE2EDuration="2.125287065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:01.122877917 +0000 UTC m=+3649.230373828" watchObservedRunningTime="2026-10-30 21:37:01.125287065 +0000 UTC m=+3649.232782976" Oct 30 21:37:01.144746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.144700 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-grpc\"" Oct 30 21:37:01.144972 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.144788 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-http\"" Oct 30 21:37:01.150651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.150618 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:01.236280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.236202 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:01.240188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.240159 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:01.322359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.322323 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:01.325695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.325673 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.333886 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.333860 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.333982 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.333914 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.334021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.333980 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.334172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.334146 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.334212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.334199 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.334248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.334240 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njpz4\" (UniqueName: \"kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.350973 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.350941 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 21:37:01.360593 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.360552 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 30 21:37:01.386273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.386240 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:01.411561 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.411531 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 30 21:37:01.411723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.411629 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 30 21:37:01.435124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435088 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435146 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435167 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435211 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435232 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435250 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpz4\" (UniqueName: \"kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435992 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435929 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.435992 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.435978 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.437679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.437651 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.437820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.437796 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.438021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.438000 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.496942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.496863 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpz4\" (UniqueName: \"kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:01.536623 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.536588 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.536778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.536631 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.536778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.536657 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.536778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.536717 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.536898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.536778 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv2v\" (UniqueName: \"kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.536898 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.536816 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.637531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.637495 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.637749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.637545 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.637749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.637586 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.637749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.637610 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.637749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.637640 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv2v\" (UniqueName: \"kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.637967 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.637780 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.638459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.638431 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.638605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.638586 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.640126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.640096 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.640275 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.640181 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.640518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.640499 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.701540 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.701509 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv2v\" (UniqueName: \"kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:01.800052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.799968 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb"] Oct 30 21:37:01.804647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.804625 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:01.857107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.857060 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-2lrnh\"" Oct 30 21:37:01.857303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.857137 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 21:37:01.860688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.860656 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 21:37:01.862295 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.862252 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb"] Oct 30 21:37:01.940544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.940512 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:01.940544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.940547 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:01.940766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:01.940702 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzqp\" (UniqueName: \"kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.016030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.015995 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7513df-1500-4aa5-9f16-d20fc090fbc9" path="/var/lib/kubelet/pods/4b7513df-1500-4aa5-9f16-d20fc090fbc9/volumes" Oct 30 21:37:02.016405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.016392 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8db61d-594f-4273-b9bf-228e1f011484" path="/var/lib/kubelet/pods/9a8db61d-594f-4273-b9bf-228e1f011484/volumes" Oct 30 21:37:02.016793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.016780 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bb886e-4838-4858-86c6-6ea7028e5978" path="/var/lib/kubelet/pods/b0bb886e-4838-4858-86c6-6ea7028e5978/volumes" Oct 30 21:37:02.041444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.041389 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzqp\" (UniqueName: \"kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.041607 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.041568 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.041693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.041607 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.042306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.042279 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.044091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.044061 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.121469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.121368 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzqp\" (UniqueName: \"kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp\") pod \"data-science-usage-logs-collector-8666d99d79-bwkvb\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.417460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.417386 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:02.792243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.792148 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb"] Oct 30 21:37:02.795351 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:37:02.795318 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7d17cc_d3d5_42b7_878a_05e996f8a28c.slice/crio-ed0c31052f90645f7651efbf5e1712c7b17a227702dbce7530086e448b05765f WatchSource:0}: Error finding container ed0c31052f90645f7651efbf5e1712c7b17a227702dbce7530086e448b05765f: Status 404 returned error can't find the container with id ed0c31052f90645f7651efbf5e1712c7b17a227702dbce7530086e448b05765f Oct 30 21:37:02.797723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.797681 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:02.797801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.797764 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:02.969347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.969309 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" event={"ID":"7d7d17cc-d3d5-42b7-878a-05e996f8a28c","Type":"ContainerStarted","Data":"033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617"} Oct 30 21:37:02.969523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:02.969369 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" event={"ID":"7d7d17cc-d3d5-42b7-878a-05e996f8a28c","Type":"ContainerStarted","Data":"ed0c31052f90645f7651efbf5e1712c7b17a227702dbce7530086e448b05765f"} Oct 30 21:37:03.086281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:03.086165 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" podStartSLOduration=2.086146314 podStartE2EDuration="2.086146314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:03.0856551 +0000 UTC m=+3651.193151010" watchObservedRunningTime="2026-10-30 21:37:03.086146314 +0000 UTC m=+3651.193642229" Oct 30 21:37:06.914138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:06.914083 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.096697 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.096652 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:07.126970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.126930 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:07.131133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.131105 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.145293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.145260 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5"] Oct 30 21:37:07.145488 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.145456 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" podUID="15a8d404-235a-4426-af60-1183867f77ad" containerName="loki-querier" containerID="cri-o://cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847" gracePeriod=30 Oct 30 21:37:07.147858 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:07.147832 4988 csi_plugin.go:220] "Failed to get volume attachment" err="volumeattachments.storage.k8s.io \"csi-3dfbb34f83e37b8263868048616466720a2a70016cd4946f13909e2269e5fd38\" is forbidden: User \"system:node:ip-10-0-141-115.us-west-1.compute.internal\" cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\" at the cluster scope: no relationship found between node 'ip-10-0-141-115.us-west-1.compute.internal' and this object" attachmentName="csi-3dfbb34f83e37b8263868048616466720a2a70016cd4946f13909e2269e5fd38" Oct 30 21:37:07.147946 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:07.147872 4988 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[storage], unattached volumes=[storage], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podUID="216347fd-da45-4ea0-b897-778ffdbbe528" Oct 30 21:37:07.153730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.153704 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8"] Oct 30 21:37:07.153907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.153886 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="gateway" containerID="cri-o://fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1" gracePeriod=30 Oct 30 21:37:07.153955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.153909 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="opa" containerID="cri-o://9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9" gracePeriod=30 Oct 30 21:37:07.183614 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:07.183570 4988 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[storage], unattached volumes=[storage], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="d7ce4cfc-604d-47f1-9245-234d29fb0a09" Oct 30 21:37:07.292171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.292125 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.579294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.579206 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb"] Oct 30 21:37:07.579460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.579394 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" podUID="7d7d17cc-d3d5-42b7-878a-05e996f8a28c" containerName="otc-container" containerID="cri-o://033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617" gracePeriod=30 Oct 30 21:37:07.609095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.609034 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.757243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.757202 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:07.832625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.832565 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:07.993256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.993221 4988 generic.go:356] "Generic (PLEG): container finished" podID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerID="9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9" exitCode=0 Oct 30 21:37:07.993688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.993274 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" event={"ID":"a1931617-7e55-450f-9c2d-2e058fc3b23e","Type":"ContainerDied","Data":"9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9"} Oct 30 21:37:07.994672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994645 4988 generic.go:356] "Generic (PLEG): container finished" podID="7d7d17cc-d3d5-42b7-878a-05e996f8a28c" containerID="033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617" exitCode=0 Oct 30 21:37:07.994795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994685 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" Oct 30 21:37:07.994795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994700 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:07.994795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994733 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" event={"ID":"7d7d17cc-d3d5-42b7-878a-05e996f8a28c","Type":"ContainerDied","Data":"033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617"} Oct 30 21:37:07.994795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994776 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb" event={"ID":"7d7d17cc-d3d5-42b7-878a-05e996f8a28c","Type":"ContainerDied","Data":"ed0c31052f90645f7651efbf5e1712c7b17a227702dbce7530086e448b05765f"} Oct 30 21:37:07.994926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994803 4988 scope.go:122] "RemoveContainer" containerID="033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617" Oct 30 21:37:07.994926 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.994819 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:07.995386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.995369 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp\" (UniqueName: \"kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp\") pod \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " Oct 30 21:37:07.995516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.995401 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal\") pod \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " Oct 30 21:37:07.995516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.995437 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs\") pod \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\" (UID: \"7d7d17cc-d3d5-42b7-878a-05e996f8a28c\") " Oct 30 21:37:07.995806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.995784 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal" pod "7d7d17cc-d3d5-42b7-878a-05e996f8a28c" (UID: "7d7d17cc-d3d5-42b7-878a-05e996f8a28c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:07.997555 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.997532 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp" pod "7d7d17cc-d3d5-42b7-878a-05e996f8a28c" (UID: "7d7d17cc-d3d5-42b7-878a-05e996f8a28c"). InnerVolumeSpecName "kube-api-access-swzqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:07.997634 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:07.997555 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs" pod "7d7d17cc-d3d5-42b7-878a-05e996f8a28c" (UID: "7d7d17cc-d3d5-42b7-878a-05e996f8a28c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.009186 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.009157 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.013718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.013694 4988 scope.go:122] "RemoveContainer" containerID="033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617" Oct 30 21:37:08.014191 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:08.014159 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617\": container with ID starting with 033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617 not found: ID does not exist" containerID="033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617" Oct 30 21:37:08.014300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.014202 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617"} err="failed to get container status \"033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617\": rpc error: code = NotFound desc = could not find container \"033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617\": container with ID starting with 033b8c35e2e5d2d340484ec59c7c24810f09d09415b23afad09a2a5440bf1617 not found: ID does not exist" Oct 30 21:37:08.017764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.017744 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:08.096759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096660 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http\") pod \"216347fd-da45-4ea0-b897-778ffdbbe528\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " Oct 30 21:37:08.096759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096710 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config\") pod \"216347fd-da45-4ea0-b897-778ffdbbe528\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " Oct 30 21:37:08.096759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096728 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4\" (UniqueName: \"kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4\") pod \"216347fd-da45-4ea0-b897-778ffdbbe528\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " Oct 30 21:37:08.096759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096747 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096773 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle\") pod \"216347fd-da45-4ea0-b897-778ffdbbe528\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096804 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v\" (UniqueName: \"kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096833 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096888 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"216347fd-da45-4ea0-b897-778ffdbbe528\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096908 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096927 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.096980 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret\") pod \"216347fd-da45-4ea0-b897-778ffdbbe528\" (UID: \"216347fd-da45-4ea0-b897-778ffdbbe528\") " Oct 30 21:37:08.097102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097015 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097168 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097152 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle" pod "216347fd-da45-4ea0-b897-778ffdbbe528" (UID: "216347fd-da45-4ea0-b897-778ffdbbe528"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097341 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097387 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097441 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzqp\" (UniqueName: \"kubernetes.io/projected/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-kube-api-access-swzqp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097458 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.097554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097475 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d7d17cc-d3d5-42b7-878a-05e996f8a28c-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.097924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097725 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.097924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.097742 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config" pod "216347fd-da45-4ea0-b897-778ffdbbe528" (UID: "216347fd-da45-4ea0-b897-778ffdbbe528"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.099807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.099766 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "kube-api-access-slv2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:08.099932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.099864 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4" pod "216347fd-da45-4ea0-b897-778ffdbbe528" (UID: "216347fd-da45-4ea0-b897-778ffdbbe528"). InnerVolumeSpecName "kube-api-access-njpz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:08.099932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.099879 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.099932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.099884 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc" pod "216347fd-da45-4ea0-b897-778ffdbbe528" (UID: "216347fd-da45-4ea0-b897-778ffdbbe528"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.099932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.099856 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.100126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.100051 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http" pod "216347fd-da45-4ea0-b897-778ffdbbe528" (UID: "216347fd-da45-4ea0-b897-778ffdbbe528"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.100126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.100076 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.101049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.101031 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret" pod "216347fd-da45-4ea0-b897-778ffdbbe528" (UID: "216347fd-da45-4ea0-b897-778ffdbbe528"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.198262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198219 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198252 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216347fd-da45-4ea0-b897-778ffdbbe528-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198269 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njpz4\" (UniqueName: \"kubernetes.io/projected/216347fd-da45-4ea0-b897-778ffdbbe528-kube-api-access-njpz4\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198283 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198297 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv2v\" (UniqueName: \"kubernetes.io/projected/d7ce4cfc-604d-47f1-9245-234d29fb0a09-kube-api-access-slv2v\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198310 4988 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-test-loki-lifecycle-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198323 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-data-science-usage-logs-lokistack-index-gateway-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198335 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198347 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198359 4988 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/216347fd-da45-4ea0-b897-778ffdbbe528-test-loki-lifecycle-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.198570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.198372 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/d7ce4cfc-604d-47f1-9245-234d29fb0a09-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.201981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.201952 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.298739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.298706 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.298907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.298846 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.350618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.350540 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") DevicePath \"csi-155d7e76a00fba862a255a008b4646e5e30bae03a1ce3c419eed77bd55a26b19\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.397627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.397598 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:37:08.410491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.410457 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb"] Oct 30 21:37:08.416865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.416838 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-8666d99d79-bwkvb"] Oct 30 21:37:08.489296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.489259 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fdd263e80916c99f6ceb9f70acac6dacd6dc4d7bb485176bf2f162ed785207a5/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.500421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500372 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500445 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500471 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500562 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500610 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500649 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500687 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500734 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:08.500977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.500820 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.501075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.501040 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.503108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.503079 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "kube-api-access-vx65f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:08.503108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.503080 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:08.503258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.503120 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:08.503339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.503325 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:08.505671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.505650 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.506711 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:08.506693 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants podName:a1931617-7e55-450f-9c2d-2e058fc3b23e nodeName:}" failed. No retries permitted until 2026-10-30 21:37:09.006676444 +0000 UTC m=+3657.114172350 (durationBeforeRetry 500ms). Error: error cleaning subPath mounts for volume "kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants" (UniqueName: "kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants") pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e") : error deleting /var/lib/kubelet/pods/a1931617-7e55-450f-9c2d-2e058fc3b23e/volume-subpaths: remove /var/lib/kubelet/pods/a1931617-7e55-450f-9c2d-2e058fc3b23e/volume-subpaths: no such file or directory Oct 30 21:37:08.506881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.506866 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:08.553878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.553847 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:08.601610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.601525 4988 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.601610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.601558 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.601610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.601568 4988 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/a1931617-7e55-450f-9c2d-2e058fc3b23e-lokistack-gateway\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.601610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.601577 4988 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-tls-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.601610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.601587 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.601610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.601597 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx65f\" (UniqueName: \"kubernetes.io/projected/a1931617-7e55-450f-9c2d-2e058fc3b23e-kube-api-access-vx65f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:08.701912 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.701877 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") pod \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\" (UID: \"d7ce4cfc-604d-47f1-9245-234d29fb0a09\") " Oct 30 21:37:08.704493 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.704464 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f" pod "d7ce4cfc-604d-47f1-9245-234d29fb0a09" (UID: "d7ce4cfc-604d-47f1-9245-234d29fb0a09"). InnerVolumeSpecName "pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:37:08.803253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.803212 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:37:08.818382 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.818359 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:37:08.904527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:08.904491 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-4a867896-64e1-4d28-a8f2-4f1338cf57d1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bb8949dd735e3b3f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-155d7e76a00fba862a255a008b4646e5e30bae03a1ce3c419eed77bd55a26b19\"" Oct 30 21:37:09.000494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.000461 4988 generic.go:356] "Generic (PLEG): container finished" podID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerID="fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1" exitCode=0 Oct 30 21:37:09.000959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.000522 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" Oct 30 21:37:09.000959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.000553 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" event={"ID":"a1931617-7e55-450f-9c2d-2e058fc3b23e","Type":"ContainerDied","Data":"fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1"} Oct 30 21:37:09.000959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.000605 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8" event={"ID":"a1931617-7e55-450f-9c2d-2e058fc3b23e","Type":"ContainerDied","Data":"d26dfab102996958e3035cbd99fadc852d548e8b4e9a4f972687099790a2733c"} Oct 30 21:37:09.000959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.000623 4988 scope.go:122] "RemoveContainer" containerID="9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9" Oct 30 21:37:09.001621 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.001605 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:09.001684 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.001623 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 21:37:09.010167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.010134 4988 scope.go:122] "RemoveContainer" containerID="fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1" Oct 30 21:37:09.018506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.018476 4988 scope.go:122] "RemoveContainer" containerID="9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9" Oct 30 21:37:09.018799 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:09.018779 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9\": container with ID starting with 9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9 not found: ID does not exist" containerID="9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9" Oct 30 21:37:09.018879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.018835 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9"} err="failed to get container status \"9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9\": rpc error: code = NotFound desc = could not find container \"9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9\": container with ID starting with 9cb913c1325bf4cee22cbe495b3e2dc36f634b7972a912198bac08ffec5f37b9 not found: ID does not exist" Oct 30 21:37:09.018879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.018862 4988 scope.go:122] "RemoveContainer" containerID="fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1" Oct 30 21:37:09.019112 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:09.019094 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1\": container with ID starting with fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1 not found: ID does not exist" containerID="fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1" Oct 30 21:37:09.019153 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.019119 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1"} err="failed to get container status \"fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1\": rpc error: code = NotFound desc = could not find container \"fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1\": container with ID starting with fd6779bc6b65c01eb4361f8da485bcd9bfa5176f5a9d610d145e00c55e0df4d1 not found: ID does not exist" Oct 30 21:37:09.105924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.105880 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\") pod \"a1931617-7e55-450f-9c2d-2e058fc3b23e\" (UID: \"a1931617-7e55-450f-9c2d-2e058fc3b23e\") " Oct 30 21:37:09.108038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.108008 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants" pod "a1931617-7e55-450f-9c2d-2e058fc3b23e" (UID: "a1931617-7e55-450f-9c2d-2e058fc3b23e"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:09.207607 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.207568 4988 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/a1931617-7e55-450f-9c2d-2e058fc3b23e-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:09.235007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.234974 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:09.240074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.240049 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:09.414610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.414530 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:09.418715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.418688 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 21:37:09.528220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.528186 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8"] Oct 30 21:37:09.532465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:09.532438 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-55c7794f8d-mswd8"] Oct 30 21:37:10.015189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.015157 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216347fd-da45-4ea0-b897-778ffdbbe528" path="/var/lib/kubelet/pods/216347fd-da45-4ea0-b897-778ffdbbe528/volumes" Oct 30 21:37:10.015578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.015447 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7d17cc-d3d5-42b7-878a-05e996f8a28c" path="/var/lib/kubelet/pods/7d7d17cc-d3d5-42b7-878a-05e996f8a28c/volumes" Oct 30 21:37:10.015797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.015784 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" path="/var/lib/kubelet/pods/a1931617-7e55-450f-9c2d-2e058fc3b23e/volumes" Oct 30 21:37:10.016471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.016457 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ce4cfc-604d-47f1-9245-234d29fb0a09" path="/var/lib/kubelet/pods/d7ce4cfc-604d-47f1-9245-234d29fb0a09/volumes" Oct 30 21:37:10.400098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.400072 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:37:10.518355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.518321 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http\") pod \"15a8d404-235a-4426-af60-1183867f77ad\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " Oct 30 21:37:10.518553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.518405 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f\" (UniqueName: \"kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f\") pod \"15a8d404-235a-4426-af60-1183867f77ad\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " Oct 30 21:37:10.518762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.518737 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle\") pod \"15a8d404-235a-4426-af60-1183867f77ad\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " Oct 30 21:37:10.518865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.518789 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc\") pod \"15a8d404-235a-4426-af60-1183867f77ad\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " Oct 30 21:37:10.518865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.518852 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret\") pod \"15a8d404-235a-4426-af60-1183867f77ad\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " Oct 30 21:37:10.518963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.518895 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config\") pod \"15a8d404-235a-4426-af60-1183867f77ad\" (UID: \"15a8d404-235a-4426-af60-1183867f77ad\") " Oct 30 21:37:10.519288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.519148 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle" pod "15a8d404-235a-4426-af60-1183867f77ad" (UID: "15a8d404-235a-4426-af60-1183867f77ad"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:10.519509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.519465 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config" pod "15a8d404-235a-4426-af60-1183867f77ad" (UID: "15a8d404-235a-4426-af60-1183867f77ad"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:10.521508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.521472 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc" pod "15a8d404-235a-4426-af60-1183867f77ad" (UID: "15a8d404-235a-4426-af60-1183867f77ad"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:10.521617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.521579 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret" pod "15a8d404-235a-4426-af60-1183867f77ad" (UID: "15a8d404-235a-4426-af60-1183867f77ad"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:10.521708 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.521687 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f" pod "15a8d404-235a-4426-af60-1183867f77ad" (UID: "15a8d404-235a-4426-af60-1183867f77ad"). InnerVolumeSpecName "kube-api-access-mc79f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:10.522000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.521984 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http" pod "15a8d404-235a-4426-af60-1183867f77ad" (UID: "15a8d404-235a-4426-af60-1183867f77ad"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:10.619991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.619911 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:10.619991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.619939 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc79f\" (UniqueName: \"kubernetes.io/projected/15a8d404-235a-4426-af60-1183867f77ad-kube-api-access-mc79f\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:10.619991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.619951 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:10.619991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.619960 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:10.619991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.619970 4988 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/15a8d404-235a-4426-af60-1183867f77ad-test-loki-lifecycle-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:10.619991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:10.619980 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15a8d404-235a-4426-af60-1183867f77ad-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:11.012911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.012872 4988 generic.go:356] "Generic (PLEG): container finished" podID="15a8d404-235a-4426-af60-1183867f77ad" containerID="cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847" exitCode=1 Oct 30 21:37:11.013097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.012932 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" Oct 30 21:37:11.013097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.012955 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" event={"ID":"15a8d404-235a-4426-af60-1183867f77ad","Type":"ContainerDied","Data":"cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847"} Oct 30 21:37:11.013097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.012995 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5" event={"ID":"15a8d404-235a-4426-af60-1183867f77ad","Type":"ContainerDied","Data":"474506cb9be6fa1cc028dbf6198443def268c0a15fbd2589f7f2d1daeea9aca8"} Oct 30 21:37:11.013097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.013013 4988 scope.go:122] "RemoveContainer" containerID="cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847" Oct 30 21:37:11.022163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.022109 4988 scope.go:122] "RemoveContainer" containerID="cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847" Oct 30 21:37:11.022488 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:11.022437 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847\": container with ID starting with cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847 not found: ID does not exist" containerID="cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847" Oct 30 21:37:11.022488 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.022471 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847"} err="failed to get container status \"cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847\": rpc error: code = NotFound desc = could not find container \"cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847\": container with ID starting with cd8bb0a52b67009674365ff22cabf49794cb5cf85f2d26c35d96e6dd523cb847 not found: ID does not exist" Oct 30 21:37:11.177515 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.177479 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5"] Oct 30 21:37:11.227054 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:11.227020 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-2gtn5"] Oct 30 21:37:12.016688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:12.016657 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a8d404-235a-4426-af60-1183867f77ad" path="/var/lib/kubelet/pods/15a8d404-235a-4426-af60-1183867f77ad/volumes" Oct 30 21:37:14.727458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727405 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5"] Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727728 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="gateway" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727739 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="gateway" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727752 4988 cpu_manager.go:414] "containerMap: removing container" podUID="7d7d17cc-d3d5-42b7-878a-05e996f8a28c" containerName="otc-container" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727757 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7d17cc-d3d5-42b7-878a-05e996f8a28c" containerName="otc-container" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727768 4988 cpu_manager.go:414] "containerMap: removing container" podUID="15a8d404-235a-4426-af60-1183867f77ad" containerName="loki-querier" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727773 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a8d404-235a-4426-af60-1183867f77ad" containerName="loki-querier" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727785 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="opa" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727790 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="opa" Oct 30 21:37:14.727854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727858 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="gateway" Oct 30 21:37:14.728108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727868 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7d17cc-d3d5-42b7-878a-05e996f8a28c" containerName="otc-container" Oct 30 21:37:14.728108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727875 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a8d404-235a-4426-af60-1183867f77ad" containerName="loki-querier" Oct 30 21:37:14.728108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.727883 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1931617-7e55-450f-9c2d-2e058fc3b23e" containerName="opa" Oct 30 21:37:14.732719 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.732690 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.766741 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.766692 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:14.786064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.786038 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-http\"" Oct 30 21:37:14.788347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.788326 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-c4bmf\"" Oct 30 21:37:14.835370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.835334 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:37:14.835646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.835631 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 30 21:37:14.835732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.835677 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:37:14.837813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.837792 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 30 21:37:14.839329 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.839310 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5"] Oct 30 21:37:14.862640 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.862607 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wk9\" (UniqueName: \"kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.862732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.862652 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.862732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.862715 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.862822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.862735 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.862870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.862831 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.886590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.886563 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-grpc\"" Oct 30 21:37:14.963646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.963605 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wk9\" (UniqueName: \"kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.963646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.963649 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.963881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.963676 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.963881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.963696 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.963881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.963716 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.964489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.964462 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.964691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.964572 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.964691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.964693 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:14.966237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.966218 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:14.966329 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:14.966278 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:15.025026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.024950 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wk9\" (UniqueName: \"kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:15.044070 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.044037 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:15.112530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.112492 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.192882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.192842 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns"] Oct 30 21:37:15.197076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.197058 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.249787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.249752 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 21:37:15.250007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.249973 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 30 21:37:15.250007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.249976 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 30 21:37:15.256145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.256117 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns"] Oct 30 21:37:15.259087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.259053 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.366943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.366848 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.366943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.366896 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.367133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.367009 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vqzq\" (UniqueName: \"kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.367133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.367095 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.367133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.367127 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.367227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.367204 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.409943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.409902 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.468066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468019 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.468237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468082 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.468237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468134 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.468237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468194 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.468237 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468221 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.468490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468261 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vqzq\" (UniqueName: \"kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.469007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.468930 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.471231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.471207 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5"] Oct 30 21:37:15.471356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.471319 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.471356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.471335 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.471578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.471559 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.472643 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.472627 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.475817 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:37:15.475790 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03cb25f0_cc99_4cfa_9cef_a8d559d8b79a.slice/crio-9444e1a712b1ef7d3bf4911dcebaee43f7c866adfa6ca56703bbcae2855df897 WatchSource:0}: Error finding container 9444e1a712b1ef7d3bf4911dcebaee43f7c866adfa6ca56703bbcae2855df897: Status 404 returned error can't find the container with id 9444e1a712b1ef7d3bf4911dcebaee43f7c866adfa6ca56703bbcae2855df897 Oct 30 21:37:15.529730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.529706 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vqzq\" (UniqueName: \"kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq\") pod \"data-science-usage-logs-lokistack-querier-c94bb499b-jpkns\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.562030 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.561996 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.710206 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.710168 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.808135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.808077 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:15.842164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.842130 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:15.845562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.845542 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:15.879560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.877850 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:15.898301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.898270 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-http\"" Oct 30 21:37:15.898301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.898270 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-grpc\"" Oct 30 21:37:15.905181 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:15.905149 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:16.027891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.027793 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.034173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.034147 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" event={"ID":"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a","Type":"ContainerStarted","Data":"67910180fd20a7949b8a69693a3d0ecf6bef1bfdce42addfc4023284beffe394"} Oct 30 21:37:16.034320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.034182 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" event={"ID":"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a","Type":"ContainerStarted","Data":"9444e1a712b1ef7d3bf4911dcebaee43f7c866adfa6ca56703bbcae2855df897"} Oct 30 21:37:16.034320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.034278 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:16.073814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.073767 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.073974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.073894 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.073974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.073954 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.074049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.073993 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.074049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.074042 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.074118 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.074067 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lh2p\" (UniqueName: \"kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.152116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.152066 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" podStartSLOduration=2.152051611 podStartE2EDuration="2.152051611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:16.151112138 +0000 UTC m=+3664.258608090" watchObservedRunningTime="2026-10-30 21:37:16.152051611 +0000 UTC m=+3664.259547523" Oct 30 21:37:16.174681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.174643 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.174899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.174694 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.174899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.174737 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.174899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.174771 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.174899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.174797 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lh2p\" (UniqueName: \"kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.174899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.174829 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.175526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.175492 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.175776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.175747 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.177053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.177025 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.177742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.177718 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.177830 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.177780 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.177938 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.177919 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.191956 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.191929 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns"] Oct 30 21:37:16.195169 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:37:16.195143 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027ed42c_0653_43a4_a5c3_79766a89ed09.slice/crio-e528a716d7d06bb331ca53d3f84c706cff56719760ebbeb41c7f48ba84cb5cdf WatchSource:0}: Error finding container e528a716d7d06bb331ca53d3f84c706cff56719760ebbeb41c7f48ba84cb5cdf: Status 404 returned error can't find the container with id e528a716d7d06bb331ca53d3f84c706cff56719760ebbeb41c7f48ba84cb5cdf Oct 30 21:37:16.236002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.235972 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lh2p\" (UniqueName: \"kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:16.382929 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.382898 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l"] Oct 30 21:37:16.386870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.386843 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.406951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.406915 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.449474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.449439 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l"] Oct 30 21:37:16.450227 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.450203 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 30 21:37:16.450363 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.450233 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 21:37:16.463178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.463140 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 30 21:37:16.463356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.463188 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 21:37:16.463356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.463222 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-jjzn8\"" Oct 30 21:37:16.463356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.463298 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 30 21:37:16.477642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477608 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477649 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477734 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477757 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477754 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6t8\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477814 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477836 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477859 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.477975 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.477879 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579269 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579307 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579461 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579505 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579561 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579586 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579642 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.579816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.579673 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6t8\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.580307 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.580285 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.580730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.580706 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.580786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.580736 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.582027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.582006 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.582104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.582013 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.582249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.582233 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.582287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.582247 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.641041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.640964 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6t8\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8\") pod \"data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:16.660527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.660462 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:16.702956 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:16.702926 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:17.047344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:17.047246 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" event={"ID":"027ed42c-0653-43a4-a5c3-79766a89ed09","Type":"ContainerStarted","Data":"f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef"} Oct 30 21:37:17.047344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:17.047296 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" event={"ID":"027ed42c-0653-43a4-a5c3-79766a89ed09","Type":"ContainerStarted","Data":"e528a716d7d06bb331ca53d3f84c706cff56719760ebbeb41c7f48ba84cb5cdf"} Oct 30 21:37:17.133357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:17.133323 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l"] Oct 30 21:37:17.137029 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:37:17.137001 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30010d8b_8724_4f24_b3fa_00e280e2b734.slice/crio-e5cd4f6aefd8d37842cc442465c2f1684f9319b56d480c4d14a540011af7ba76 WatchSource:0}: Error finding container e5cd4f6aefd8d37842cc442465c2f1684f9319b56d480c4d14a540011af7ba76: Status 404 returned error can't find the container with id e5cd4f6aefd8d37842cc442465c2f1684f9319b56d480c4d14a540011af7ba76 Oct 30 21:37:17.166857 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:17.165500 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" podStartSLOduration=2.165482842 podStartE2EDuration="2.165482842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:17.165069938 +0000 UTC m=+3665.272565850" watchObservedRunningTime="2026-10-30 21:37:17.165482842 +0000 UTC m=+3665.272978757" Oct 30 21:37:18.055125 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.055084 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" event={"ID":"30010d8b-8724-4f24-b3fa-00e280e2b734","Type":"ContainerStarted","Data":"f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82"} Oct 30 21:37:18.055841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.055137 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" event={"ID":"30010d8b-8724-4f24-b3fa-00e280e2b734","Type":"ContainerStarted","Data":"237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006"} Oct 30 21:37:18.055841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.055152 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" event={"ID":"30010d8b-8724-4f24-b3fa-00e280e2b734","Type":"ContainerStarted","Data":"e5cd4f6aefd8d37842cc442465c2f1684f9319b56d480c4d14a540011af7ba76"} Oct 30 21:37:18.055841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.055201 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:18.176205 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.176151 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" podStartSLOduration=2.176137071 podStartE2EDuration="2.176137071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:18.175773561 +0000 UTC m=+3666.283269472" watchObservedRunningTime="2026-10-30 21:37:18.176137071 +0000 UTC m=+3666.283632987" Oct 30 21:37:18.498693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.498651 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:18.603294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.603259 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:18.701181 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.701141 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:18.701360 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.701283 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:18.753586 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.753509 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") DevicePath \"csi-3dfbb34f83e37b8263868048616466720a2a70016cd4946f13909e2269e5fd38\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:18.892792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.892753 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/16847625ef5b169474e6c7fa0db03454ec63f6f80fcb0eec5f5f82909c6eec52/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:18.956566 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:18.956531 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:19.060562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:19.060472 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:19.060562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:19.060525 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:19.067768 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:19.067742 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:19.070522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:19.070500 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:37:19.157345 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:19.157303 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:19.582288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:19.582256 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:37:19.585585 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:37:19.585559 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e0bf409_6d24_498e_9edc_4cd3bf28774f.slice/crio-03e90ecf5d6574d0eb226c46e397e447b414639a3092022a0bd024fa50c153cf WatchSource:0}: Error finding container 03e90ecf5d6574d0eb226c46e397e447b414639a3092022a0bd024fa50c153cf: Status 404 returned error can't find the container with id 03e90ecf5d6574d0eb226c46e397e447b414639a3092022a0bd024fa50c153cf Oct 30 21:37:20.066571 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:20.066531 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"8e0bf409-6d24-498e-9edc-4cd3bf28774f","Type":"ContainerStarted","Data":"88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf"} Oct 30 21:37:20.066978 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:20.066585 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"8e0bf409-6d24-498e-9edc-4cd3bf28774f","Type":"ContainerStarted","Data":"03e90ecf5d6574d0eb226c46e397e447b414639a3092022a0bd024fa50c153cf"} Oct 30 21:37:20.066978 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:20.066731 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:20.185927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:20.185868 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podStartSLOduration=5.185852461 podStartE2EDuration="5.185852461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:20.184649529 +0000 UTC m=+3668.292145446" watchObservedRunningTime="2026-10-30 21:37:20.185852461 +0000 UTC m=+3668.293348371" Oct 30 21:37:22.383892 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:22.383858 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:37:26.394567 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:26.394530 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:37:37.053465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:37.053434 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:37:39.066667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:39.066639 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:37:41.077955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:37:41.077923 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:37:56.393986 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:37:56.393948 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:38:13.129687 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:13.129654 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:22.678219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:22.678109 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:25.175580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:25.175534 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:26.394730 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:26.394682 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:38:31.079034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.078995 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.106075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.106034 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf"] Oct 30 21:38:31.109689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.109668 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.147691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.147646 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.147885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.147711 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.147885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.147814 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.168134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.168097 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf"] Oct 30 21:38:31.196290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.196253 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 21:38:31.211754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.211707 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-z9prh\"" Oct 30 21:38:31.211936 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.211771 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 21:38:31.233962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.233925 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.249039 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.249008 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.249229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.249055 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.249229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.249093 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.249402 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:31.249232 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 21:38:31.249402 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:31.249304 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs podName:e155203f-c788-45d7-a47a-03f32c47adba nodeName:}" failed. No retries permitted until 2026-10-30 21:38:31.749283892 +0000 UTC m=+3739.856779798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs") pod "data-science-usage-logs-collector-86f7998849-zw7mf" (UID: "e155203f-c788-45d7-a47a-03f32c47adba") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 30 21:38:31.249688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.249671 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.312846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.312808 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.510792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.510748 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:31.753911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.753875 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:31.756254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:31.756225 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\") pod \"data-science-usage-logs-collector-86f7998849-zw7mf\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:32.024842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:32.024793 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:32.403459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:32.403425 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf"] Oct 30 21:38:32.406589 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:38:32.406556 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode155203f_c788_45d7_a47a_03f32c47adba.slice/crio-2d2fd24b8f8f20c7c30f7783da1536ea9e13eb0a7bd91818ada4f0d655f2d46d WatchSource:0}: Error finding container 2d2fd24b8f8f20c7c30f7783da1536ea9e13eb0a7bd91818ada4f0d655f2d46d: Status 404 returned error can't find the container with id 2d2fd24b8f8f20c7c30f7783da1536ea9e13eb0a7bd91818ada4f0d655f2d46d Oct 30 21:38:32.408633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:32.408595 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:32.408737 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:32.408670 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:33.391746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:33.391708 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" event={"ID":"e155203f-c788-45d7-a47a-03f32c47adba","Type":"ContainerStarted","Data":"73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6"} Oct 30 21:38:33.391966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:33.391802 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" event={"ID":"e155203f-c788-45d7-a47a-03f32c47adba","Type":"ContainerStarted","Data":"2d2fd24b8f8f20c7c30f7783da1536ea9e13eb0a7bd91818ada4f0d655f2d46d"} Oct 30 21:38:33.509849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:33.509791 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" podStartSLOduration=2.509775219 podStartE2EDuration="2.509775219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:33.508540592 +0000 UTC m=+3741.616036503" watchObservedRunningTime="2026-10-30 21:38:33.509775219 +0000 UTC m=+3741.617271129" Oct 30 21:38:37.632955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.632874 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:37.720869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.720836 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:38:37.721090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.721059 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podUID="8e0bf409-6d24-498e-9edc-4cd3bf28774f" containerName="loki-index-gateway" containerID="cri-o://88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf" gracePeriod=30 Oct 30 21:38:37.765062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.765028 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns"] Oct 30 21:38:37.765269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.765243 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" podUID="027ed42c-0653-43a4-a5c3-79766a89ed09" containerName="loki-querier" containerID="cri-o://f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef" gracePeriod=30 Oct 30 21:38:37.771966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.771935 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5"] Oct 30 21:38:37.772164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.772140 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerName="loki-distributor" containerID="cri-o://67910180fd20a7949b8a69693a3d0ecf6bef1bfdce42addfc4023284beffe394" gracePeriod=30 Oct 30 21:38:37.776233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.776202 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l"] Oct 30 21:38:37.776479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.776449 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="gateway" containerID="cri-o://237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006" gracePeriod=30 Oct 30 21:38:37.776567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.776484 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="opa" containerID="cri-o://f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82" gracePeriod=30 Oct 30 21:38:37.871063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:37.871020 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.084110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.084087 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:38:38.210251 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210217 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210462 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210263 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p\" (UniqueName: \"kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210462 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210318 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210462 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210450 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210552 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210615 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210709 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:38.210771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210723 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle\") pod \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\" (UID: \"8e0bf409-6d24-498e-9edc-4cd3bf28774f\") " Oct 30 21:38:38.210997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.210974 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.211188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.211155 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:38.212967 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.212941 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "kube-api-access-9lh2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:38.213112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.213083 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.213163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.213127 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.213371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.213350 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.213852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.213832 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751" pod "8e0bf409-6d24-498e-9edc-4cd3bf28774f" (UID: "8e0bf409-6d24-498e-9edc-4cd3bf28774f"). InnerVolumeSpecName "pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:38:38.225020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.224989 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:38:38.312232 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312190 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq\" (UniqueName: \"kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq\") pod \"027ed42c-0653-43a4-a5c3-79766a89ed09\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " Oct 30 21:38:38.312232 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312238 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle\") pod \"027ed42c-0653-43a4-a5c3-79766a89ed09\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " Oct 30 21:38:38.312545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312295 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http\") pod \"027ed42c-0653-43a4-a5c3-79766a89ed09\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " Oct 30 21:38:38.312545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312346 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config\") pod \"027ed42c-0653-43a4-a5c3-79766a89ed09\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " Oct 30 21:38:38.312545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312395 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc\") pod \"027ed42c-0653-43a4-a5c3-79766a89ed09\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " Oct 30 21:38:38.312545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312428 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret\") pod \"027ed42c-0653-43a4-a5c3-79766a89ed09\" (UID: \"027ed42c-0653-43a4-a5c3-79766a89ed09\") " Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312648 4988 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-test-loki-lifecycle-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312685 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312703 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312719 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312714 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle" pod "027ed42c-0653-43a4-a5c3-79766a89ed09" (UID: "027ed42c-0653-43a4-a5c3-79766a89ed09"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312736 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/8e0bf409-6d24-498e-9edc-4cd3bf28774f-data-science-usage-logs-lokistack-index-gateway-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.312774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312751 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lh2p\" (UniqueName: \"kubernetes.io/projected/8e0bf409-6d24-498e-9edc-4cd3bf28774f-kube-api-access-9lh2p\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.313107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.312796 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config" pod "027ed42c-0653-43a4-a5c3-79766a89ed09" (UID: "027ed42c-0653-43a4-a5c3-79766a89ed09"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:38.314758 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.314726 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc" pod "027ed42c-0653-43a4-a5c3-79766a89ed09" (UID: "027ed42c-0653-43a4-a5c3-79766a89ed09"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.314870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.314845 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq" pod "027ed42c-0653-43a4-a5c3-79766a89ed09" (UID: "027ed42c-0653-43a4-a5c3-79766a89ed09"). InnerVolumeSpecName "kube-api-access-6vqzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:38.314911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.314864 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http" pod "027ed42c-0653-43a4-a5c3-79766a89ed09" (UID: "027ed42c-0653-43a4-a5c3-79766a89ed09"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.315180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.315161 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret" pod "027ed42c-0653-43a4-a5c3-79766a89ed09" (UID: "027ed42c-0653-43a4-a5c3-79766a89ed09"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.321767 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.321749 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:38:38.413342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413244 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.413342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413276 4988 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-test-loki-lifecycle-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.413342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413291 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vqzq\" (UniqueName: \"kubernetes.io/projected/027ed42c-0653-43a4-a5c3-79766a89ed09-kube-api-access-6vqzq\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.413342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413304 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.413342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413319 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/027ed42c-0653-43a4-a5c3-79766a89ed09-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.413342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413334 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-2384f338-0932-4bc9-a23a-67c67aa8f5bf\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b4268862ca575751\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-3dfbb34f83e37b8263868048616466720a2a70016cd4946f13909e2269e5fd38\"" Oct 30 21:38:38.413710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.413348 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/027ed42c-0653-43a4-a5c3-79766a89ed09-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.416049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.416020 4988 generic.go:356] "Generic (PLEG): container finished" podID="8e0bf409-6d24-498e-9edc-4cd3bf28774f" containerID="88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf" exitCode=0 Oct 30 21:38:38.416173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.416072 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 30 21:38:38.416173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.416104 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"8e0bf409-6d24-498e-9edc-4cd3bf28774f","Type":"ContainerDied","Data":"88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf"} Oct 30 21:38:38.416173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.416146 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"8e0bf409-6d24-498e-9edc-4cd3bf28774f","Type":"ContainerDied","Data":"03e90ecf5d6574d0eb226c46e397e447b414639a3092022a0bd024fa50c153cf"} Oct 30 21:38:38.416173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.416163 4988 scope.go:122] "RemoveContainer" containerID="88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf" Oct 30 21:38:38.417579 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.417549 4988 generic.go:356] "Generic (PLEG): container finished" podID="027ed42c-0653-43a4-a5c3-79766a89ed09" containerID="f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef" exitCode=0 Oct 30 21:38:38.417681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.417608 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" Oct 30 21:38:38.417681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.417630 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" event={"ID":"027ed42c-0653-43a4-a5c3-79766a89ed09","Type":"ContainerDied","Data":"f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef"} Oct 30 21:38:38.417681 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.417670 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns" event={"ID":"027ed42c-0653-43a4-a5c3-79766a89ed09","Type":"ContainerDied","Data":"e528a716d7d06bb331ca53d3f84c706cff56719760ebbeb41c7f48ba84cb5cdf"} Oct 30 21:38:38.419705 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.419680 4988 generic.go:356] "Generic (PLEG): container finished" podID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerID="f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82" exitCode=0 Oct 30 21:38:38.419806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.419743 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" event={"ID":"30010d8b-8724-4f24-b3fa-00e280e2b734","Type":"ContainerDied","Data":"f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82"} Oct 30 21:38:38.426125 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.426103 4988 scope.go:122] "RemoveContainer" containerID="88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf" Oct 30 21:38:38.426521 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:38.426490 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf\": container with ID starting with 88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf not found: ID does not exist" containerID="88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf" Oct 30 21:38:38.426587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.426537 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf"} err="failed to get container status \"88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf\": rpc error: code = NotFound desc = could not find container \"88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf\": container with ID starting with 88bda1ae70dc49f7e2ed57a04a6c3b7c0f51e6b7dadaea04c8474ac239a43bcf not found: ID does not exist" Oct 30 21:38:38.426587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.426566 4988 scope.go:122] "RemoveContainer" containerID="f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef" Oct 30 21:38:38.434915 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.434896 4988 scope.go:122] "RemoveContainer" containerID="f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef" Oct 30 21:38:38.435172 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:38.435153 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef\": container with ID starting with f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef not found: ID does not exist" containerID="f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef" Oct 30 21:38:38.435223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.435180 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef"} err="failed to get container status \"f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef\": rpc error: code = NotFound desc = could not find container \"f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef\": container with ID starting with f943c7dd8af2c8bf97ebd6923892e66e81617c5c354d95648ba5ec20038494ef not found: ID does not exist" Oct 30 21:38:38.503325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.503280 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf"] Oct 30 21:38:38.503588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.503547 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" podUID="e155203f-c788-45d7-a47a-03f32c47adba" containerName="otc-container" containerID="cri-o://73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6" gracePeriod=30 Oct 30 21:38:38.533403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.533367 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.565470 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.565438 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:38:38.570309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.570281 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 30 21:38:38.681980 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.681845 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:38:38.681980 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.681938 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns"] Oct 30 21:38:38.682439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.681998 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-c94bb499b-jpkns"] Oct 30 21:38:38.730363 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.730338 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:38.816527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.816486 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal\") pod \"e155203f-c788-45d7-a47a-03f32c47adba\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " Oct 30 21:38:38.816527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.816532 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp\") pod \"e155203f-c788-45d7-a47a-03f32c47adba\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " Oct 30 21:38:38.816730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.816580 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\") pod \"e155203f-c788-45d7-a47a-03f32c47adba\" (UID: \"e155203f-c788-45d7-a47a-03f32c47adba\") " Oct 30 21:38:38.816937 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.816911 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal" pod "e155203f-c788-45d7-a47a-03f32c47adba" (UID: "e155203f-c788-45d7-a47a-03f32c47adba"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:38.818741 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.818713 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs" pod "e155203f-c788-45d7-a47a-03f32c47adba" (UID: "e155203f-c788-45d7-a47a-03f32c47adba"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.818818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.818788 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp" pod "e155203f-c788-45d7-a47a-03f32c47adba" (UID: "e155203f-c788-45d7-a47a-03f32c47adba"). InnerVolumeSpecName "kube-api-access-gj5sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:38.917878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.917846 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e155203f-c788-45d7-a47a-03f32c47adba-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.917878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.917876 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5sp\" (UniqueName: \"kubernetes.io/projected/e155203f-c788-45d7-a47a-03f32c47adba-kube-api-access-gj5sp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:38.917878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:38.917886 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e155203f-c788-45d7-a47a-03f32c47adba-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.031553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.031525 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:38:39.119556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119513 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.119760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119566 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.119760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119605 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.119760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119665 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.119760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119694 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.119760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119746 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.120021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119781 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.120021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119811 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"30010d8b-8724-4f24-b3fa-00e280e2b734\" (UID: \"30010d8b-8724-4f24-b3fa-00e280e2b734\") " Oct 30 21:38:39.120021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.119920 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:39.120156 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.120061 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.121994 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.121955 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:39.122145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.122029 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:39.122145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.122139 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:39.122797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.122768 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "kube-api-access-xh6t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:39.125027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.124995 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:39.126178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.126157 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:39.128076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.128055 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants" pod "30010d8b-8724-4f24-b3fa-00e280e2b734" (UID: "30010d8b-8724-4f24-b3fa-00e280e2b734"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221231 4988 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-rbac\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221268 4988 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-tls-secret\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221278 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221288 4988 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/30010d8b-8724-4f24-b3fa-00e280e2b734-lokistack-gateway\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221298 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6t8\" (UniqueName: \"kubernetes.io/projected/30010d8b-8724-4f24-b3fa-00e280e2b734-kube-api-access-xh6t8\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221307 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.221325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.221317 4988 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/30010d8b-8724-4f24-b3fa-00e280e2b734-tenants\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:39.426595 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.426561 4988 generic.go:356] "Generic (PLEG): container finished" podID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerID="237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006" exitCode=0 Oct 30 21:38:39.426754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.426613 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" Oct 30 21:38:39.426754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.426648 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" event={"ID":"30010d8b-8724-4f24-b3fa-00e280e2b734","Type":"ContainerDied","Data":"237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006"} Oct 30 21:38:39.426754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.426693 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l" event={"ID":"30010d8b-8724-4f24-b3fa-00e280e2b734","Type":"ContainerDied","Data":"e5cd4f6aefd8d37842cc442465c2f1684f9319b56d480c4d14a540011af7ba76"} Oct 30 21:38:39.426754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.426717 4988 scope.go:122] "RemoveContainer" containerID="f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82" Oct 30 21:38:39.428165 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.428136 4988 generic.go:356] "Generic (PLEG): container finished" podID="e155203f-c788-45d7-a47a-03f32c47adba" containerID="73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6" exitCode=0 Oct 30 21:38:39.428284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.428174 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" Oct 30 21:38:39.428284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.428181 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" event={"ID":"e155203f-c788-45d7-a47a-03f32c47adba","Type":"ContainerDied","Data":"73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6"} Oct 30 21:38:39.428284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.428236 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf" event={"ID":"e155203f-c788-45d7-a47a-03f32c47adba","Type":"ContainerDied","Data":"2d2fd24b8f8f20c7c30f7783da1536ea9e13eb0a7bd91818ada4f0d655f2d46d"} Oct 30 21:38:39.436372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.436353 4988 scope.go:122] "RemoveContainer" containerID="237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006" Oct 30 21:38:39.444982 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.444962 4988 scope.go:122] "RemoveContainer" containerID="f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82" Oct 30 21:38:39.445272 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:39.445253 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82\": container with ID starting with f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82 not found: ID does not exist" containerID="f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82" Oct 30 21:38:39.445348 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.445280 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82"} err="failed to get container status \"f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82\": rpc error: code = NotFound desc = could not find container \"f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82\": container with ID starting with f146a60dfed0c8585e28e2dc0479c0cdc62551cdfaed4fe94dca648eb14f7e82 not found: ID does not exist" Oct 30 21:38:39.445348 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.445296 4988 scope.go:122] "RemoveContainer" containerID="237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006" Oct 30 21:38:39.445634 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:39.445617 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006\": container with ID starting with 237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006 not found: ID does not exist" containerID="237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006" Oct 30 21:38:39.445691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.445639 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006"} err="failed to get container status \"237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006\": rpc error: code = NotFound desc = could not find container \"237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006\": container with ID starting with 237bc5a1a385bfba24b5be5f65d6fd36a2f1f953a388362c55941108fdb6c006 not found: ID does not exist" Oct 30 21:38:39.445691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.445653 4988 scope.go:122] "RemoveContainer" containerID="73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6" Oct 30 21:38:39.453061 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.453042 4988 scope.go:122] "RemoveContainer" containerID="73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6" Oct 30 21:38:39.453297 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:39.453278 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6\": container with ID starting with 73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6 not found: ID does not exist" containerID="73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6" Oct 30 21:38:39.453336 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.453307 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6"} err="failed to get container status \"73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6\": rpc error: code = NotFound desc = could not find container \"73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6\": container with ID starting with 73f395e7e936b28d3c250d3cbe84d5218008d2828990e2974a9aa7d2906d30e6 not found: ID does not exist" Oct 30 21:38:39.592260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.591889 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l"] Oct 30 21:38:39.598624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.598596 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6ccc687d59-mg78l"] Oct 30 21:38:39.708800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.708765 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf"] Oct 30 21:38:39.713191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:39.713165 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-86f7998849-zw7mf"] Oct 30 21:38:40.015759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:40.015726 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027ed42c-0653-43a4-a5c3-79766a89ed09" path="/var/lib/kubelet/pods/027ed42c-0653-43a4-a5c3-79766a89ed09/volumes" Oct 30 21:38:40.016102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:40.016090 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" path="/var/lib/kubelet/pods/30010d8b-8724-4f24-b3fa-00e280e2b734/volumes" Oct 30 21:38:40.016524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:40.016512 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0bf409-6d24-498e-9edc-4cd3bf28774f" path="/var/lib/kubelet/pods/8e0bf409-6d24-498e-9edc-4cd3bf28774f/volumes" Oct 30 21:38:40.017133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:40.017120 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e155203f-c788-45d7-a47a-03f32c47adba" path="/var/lib/kubelet/pods/e155203f-c788-45d7-a47a-03f32c47adba/volumes" Oct 30 21:38:47.051351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:47.051307 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:38:56.394332 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:38:56.394296 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:38:57.052021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:57.051976 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:38:58.512499 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.512465 4988 generic.go:356] "Generic (PLEG): container finished" podID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerID="67910180fd20a7949b8a69693a3d0ecf6bef1bfdce42addfc4023284beffe394" exitCode=0 Oct 30 21:38:58.512876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.512534 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" event={"ID":"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a","Type":"ContainerDied","Data":"67910180fd20a7949b8a69693a3d0ecf6bef1bfdce42addfc4023284beffe394"} Oct 30 21:38:58.512876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.512585 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" event={"ID":"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a","Type":"ContainerDied","Data":"9444e1a712b1ef7d3bf4911dcebaee43f7c866adfa6ca56703bbcae2855df897"} Oct 30 21:38:58.512876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.512598 4988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9444e1a712b1ef7d3bf4911dcebaee43f7c866adfa6ca56703bbcae2855df897" Oct 30 21:38:58.523774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.523748 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:38:58.681503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.681462 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http\") pod \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " Oct 30 21:38:58.681503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.681502 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle\") pod \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " Oct 30 21:38:58.681775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.681568 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc\") pod \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " Oct 30 21:38:58.681775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.681591 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9\" (UniqueName: \"kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9\") pod \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " Oct 30 21:38:58.681775 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.681624 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config\") pod \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\" (UID: \"03cb25f0-cc99-4cfa-9cef-a8d559d8b79a\") " Oct 30 21:38:58.682006 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.681978 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle" pod "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" (UID: "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:58.682086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.682047 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config" pod "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" (UID: "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:58.683821 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.683791 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc" pod "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" (UID: "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:58.683821 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.683811 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http" pod "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" (UID: "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:58.683955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.683816 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9" pod "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" (UID: "03cb25f0-cc99-4cfa-9cef-a8d559d8b79a"). InnerVolumeSpecName "kube-api-access-l2wk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:58.782654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.782613 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wk9\" (UniqueName: \"kubernetes.io/projected/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-kube-api-access-l2wk9\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:58.782654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.782648 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:58.782654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.782658 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-http\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:58.782887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.782670 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:58.782887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:58.782679 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a-data-science-usage-logs-lokistack-distributor-grpc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:59.517093 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:59.517061 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5" Oct 30 21:38:59.681597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:59.681552 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5"] Oct 30 21:38:59.731110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:38:59.731078 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-rs6m5"] Oct 30 21:39:00.016341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:00.016308 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" path="/var/lib/kubelet/pods/03cb25f0-cc99-4cfa-9cef-a8d559d8b79a/volumes" Oct 30 21:39:06.705567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:06.705523 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.292720 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:16.292682 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.438576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:16.438524 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.588938 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:16.588726 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.738134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:16.738093 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:16.884875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:16.884839 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.027645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.027588 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.189099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.189016 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.337720 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.337680 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.341667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341636 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7"] Oct 30 21:39:17.341952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341940 4988 cpu_manager.go:414] "containerMap: removing container" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerName="loki-distributor" Oct 30 21:39:17.341993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341953 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerName="loki-distributor" Oct 30 21:39:17.341993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341966 4988 cpu_manager.go:414] "containerMap: removing container" podUID="027ed42c-0653-43a4-a5c3-79766a89ed09" containerName="loki-querier" Oct 30 21:39:17.341993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341972 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027ed42c-0653-43a4-a5c3-79766a89ed09" containerName="loki-querier" Oct 30 21:39:17.341993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341980 4988 cpu_manager.go:414] "containerMap: removing container" podUID="8e0bf409-6d24-498e-9edc-4cd3bf28774f" containerName="loki-index-gateway" Oct 30 21:39:17.341993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341984 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0bf409-6d24-498e-9edc-4cd3bf28774f" containerName="loki-index-gateway" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.341999 4988 cpu_manager.go:414] "containerMap: removing container" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="gateway" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342003 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="gateway" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342009 4988 cpu_manager.go:414] "containerMap: removing container" podUID="e155203f-c788-45d7-a47a-03f32c47adba" containerName="otc-container" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342014 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e155203f-c788-45d7-a47a-03f32c47adba" containerName="otc-container" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342022 4988 cpu_manager.go:414] "containerMap: removing container" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="opa" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342028 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="opa" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342076 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="027ed42c-0653-43a4-a5c3-79766a89ed09" containerName="loki-querier" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342083 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0bf409-6d24-498e-9edc-4cd3bf28774f" containerName="loki-index-gateway" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342091 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="gateway" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342096 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e155203f-c788-45d7-a47a-03f32c47adba" containerName="otc-container" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342102 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="30010d8b-8724-4f24-b3fa-00e280e2b734" containerName="opa" Oct 30 21:39:17.342145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.342109 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cb25f0-cc99-4cfa-9cef-a8d559d8b79a" containerName="loki-distributor" Oct 30 21:39:17.345273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.345252 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.398098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.398060 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:39:17.398098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.398063 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:39:17.427617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.427565 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.427617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.427613 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs9tm\" (UniqueName: \"kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.427831 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.427669 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.427831 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.427690 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.427831 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.427713 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.447581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.447493 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:39:17.447581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.447517 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:39:17.447581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.447492 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:39:17.447581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.447492 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:39:17.447581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.447557 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-rcbks\"" Oct 30 21:39:17.448990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.448965 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7"] Oct 30 21:39:17.456812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.456784 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc"] Oct 30 21:39:17.460293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.460273 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.514159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.514127 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:39:17.514364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.514332 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-2p55k\"" Oct 30 21:39:17.514459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.514390 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:39:17.518801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.518780 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc"] Oct 30 21:39:17.528891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.528860 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2dp\" (UniqueName: \"kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.529016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.528903 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.529062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529008 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529049 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.529141 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529087 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs9tm\" (UniqueName: \"kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529149 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529177 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529214 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529699 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529808 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.529887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.529872 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.531641 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.531622 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.592469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.592434 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs9tm\" (UniqueName: \"kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.630425 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.630377 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2dp\" (UniqueName: \"kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.630787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.630442 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.630787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.630469 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.631226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.631202 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.633169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.633146 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.662053 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.662019 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:39:17.692691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.692657 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2dp\" (UniqueName: \"kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp\") pod \"data-science-thanos-querier-proxy-77657866bd-7srbc\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:17.731041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.731007 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:17.777578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:17.777542 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:39:18.039553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.039476 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7"] Oct 30 21:39:18.043177 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:39:18.043151 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bb217e_6b85_4abf_8588_db4652cabe16.slice/crio-8edaa9a363fbc96535f7aca1cd7bb41240a8c15b0b07492942be04091aa871dc WatchSource:0}: Error finding container 8edaa9a363fbc96535f7aca1cd7bb41240a8c15b0b07492942be04091aa871dc: Status 404 returned error can't find the container with id 8edaa9a363fbc96535f7aca1cd7bb41240a8c15b0b07492942be04091aa871dc Oct 30 21:39:18.154439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.154400 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc"] Oct 30 21:39:18.160108 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:39:18.160078 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08701e18_d0af_45b3_a571_274982048103.slice/crio-fe99a54659616d899ef56f39d6a4a78624cc1ff701b41573ccdd6f7f9dbf6f04 WatchSource:0}: Error finding container fe99a54659616d899ef56f39d6a4a78624cc1ff701b41573ccdd6f7f9dbf6f04: Status 404 returned error can't find the container with id fe99a54659616d899ef56f39d6a4a78624cc1ff701b41573ccdd6f7f9dbf6f04 Oct 30 21:39:18.601479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.601372 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerStarted","Data":"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b"} Oct 30 21:39:18.601479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.601465 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerStarted","Data":"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276"} Oct 30 21:39:18.601977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.601494 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerStarted","Data":"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d"} Oct 30 21:39:18.601977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.601524 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerStarted","Data":"8edaa9a363fbc96535f7aca1cd7bb41240a8c15b0b07492942be04091aa871dc"} Oct 30 21:39:18.603580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.603543 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerStarted","Data":"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3"} Oct 30 21:39:18.603698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.603615 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerStarted","Data":"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7"} Oct 30 21:39:18.603698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.603650 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerStarted","Data":"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db"} Oct 30 21:39:18.603698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.603672 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerStarted","Data":"fe99a54659616d899ef56f39d6a4a78624cc1ff701b41573ccdd6f7f9dbf6f04"} Oct 30 21:39:18.837992 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.837931 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" podStartSLOduration=1.837915188 podStartE2EDuration="1.837915188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:18.720732026 +0000 UTC m=+3786.828227936" watchObservedRunningTime="2026-10-30 21:39:18.837915188 +0000 UTC m=+3786.945411132" Oct 30 21:39:18.839119 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:18.839077 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" podStartSLOduration=1.839063522 podStartE2EDuration="1.839063522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:18.837448428 +0000 UTC m=+3786.944944338" watchObservedRunningTime="2026-10-30 21:39:18.839063522 +0000 UTC m=+3786.946559432" Oct 30 21:39:19.067461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:19.067427 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:22.176463 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.176423 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:22.202676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.202638 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:39:22.206188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.206163 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.271874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.271825 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:39:22.272087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.271995 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.272087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.272045 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.272189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.272123 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.272242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.272221 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.272280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.272251 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.272280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.272273 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.308610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.308575 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-khslt\"" Oct 30 21:39:22.308852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.308829 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:39:22.308933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.308879 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:39:22.308933 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.308888 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:39:22.319311 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.319279 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:39:22.319489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.319355 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:39:22.323792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.323757 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:22.373628 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.373595 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.373823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.373656 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.373823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.373684 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.373823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.373726 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.374008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.373840 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.374008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.373878 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.374578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.374548 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.376335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.376310 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.376530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.376403 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.376591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.376559 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.376812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.376790 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.435581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.435498 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj\") pod \"data-science-collector-collector-1\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.470995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.470951 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:22.519035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.518997 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:39:22.619202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.619144 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:22.904896 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.904861 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:39:22.931068 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:39:22.931040 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcb5b2a_395e_478b_818d_6321e760cba4.slice/crio-4c7095f887152758f518461d834076d603e2cf7202bb356767aaf13d70228ba2 WatchSource:0}: Error finding container 4c7095f887152758f518461d834076d603e2cf7202bb356767aaf13d70228ba2: Status 404 returned error can't find the container with id 4c7095f887152758f518461d834076d603e2cf7202bb356767aaf13d70228ba2 Oct 30 21:39:22.933209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.933178 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:22.933299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:22.933242 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:39:23.030523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:23.030480 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:23.628715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:23.628673 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dbcb5b2a-395e-478b-818d-6321e760cba4","Type":"ContainerStarted","Data":"260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb"} Oct 30 21:39:23.629202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:23.628735 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dbcb5b2a-395e-478b-818d-6321e760cba4","Type":"ContainerStarted","Data":"4c7095f887152758f518461d834076d603e2cf7202bb356767aaf13d70228ba2"} Oct 30 21:39:23.747035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:23.746975 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.746958433 podStartE2EDuration="1.746958433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:23.746135827 +0000 UTC m=+3791.853631777" watchObservedRunningTime="2026-10-30 21:39:23.746958433 +0000 UTC m=+3791.854454343" Oct 30 21:39:26.394653 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:39:26.394613 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:39:33.104403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:33.104362 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.260591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:33.260552 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.535185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:34.535147 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.873436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:34.873312 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.035099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:35.035064 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.186505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:35.186450 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.762542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:35.762505 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.912583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:35.912546 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.955461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:35.955420 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:39:35.959443 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:35.959395 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.015082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.014999 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:39:36.018807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.018784 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.058848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.058814 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:39:36.058848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.058815 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:39:36.059085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.058853 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:39:36.059085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.058830 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-btks9\"" Oct 30 21:39:36.059085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.058861 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:39:36.060309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.060289 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:39:36.072506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.072478 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:39:36.072647 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.072633 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:39:36.072701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.072686 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:39:36.109747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.109713 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:39:36.117495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117468 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:39:36.117659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117469 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:39:36.117659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117519 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:39:36.117659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117533 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:39:36.117659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117533 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:39:36.117659 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117568 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:39:36.117833 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.117597 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jgwr8\"" Oct 30 21:39:36.184839 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.184801 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.184839 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.184842 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.184866 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.184961 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185000 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185018 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185054 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185080 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wlr\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185132 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ccs\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185153 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185172 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185240 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185303 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185441 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185458 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185475 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185529 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185547 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.185786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.185643 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.286985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.286885 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.286985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.286931 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.286985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.286951 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.286985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.286982 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287020 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287064 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287089 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287112 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287144 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wlr\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287170 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ccs\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287195 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287225 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287259 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287285 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.287342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287328 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.288008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287358 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.288008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287387 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.288008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287773 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.288008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287790 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.288008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.287853 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.288388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.288358 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.288841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.288814 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.290172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.290139 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.290287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.290249 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.290846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.290509 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.290846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.290605 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.290846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.290689 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.291605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.291212 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.291605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.291335 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.291605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.291431 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.291605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.291555 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.292715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.292696 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.293253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.293232 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.293327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.293280 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.293394 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.293373 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.293474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.293456 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.352101 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.352055 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ccs\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:36.352286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.352106 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wlr\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:36.577128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:36.577037 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:37.010308 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:37.010279 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:39:37.012930 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:39:37.012903 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44562601_163d_400f_9394_8c5d6377eadb.slice/crio-6b70fc69f3ddea0f7ba61d03c2bebb161b89602633136d5f5252c6cf69d272d2 WatchSource:0}: Error finding container 6b70fc69f3ddea0f7ba61d03c2bebb161b89602633136d5f5252c6cf69d272d2: Status 404 returned error can't find the container with id 6b70fc69f3ddea0f7ba61d03c2bebb161b89602633136d5f5252c6cf69d272d2 Oct 30 21:39:37.698307 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:37.698264 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerStarted","Data":"7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458"} Oct 30 21:39:37.698542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:37.698328 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerStarted","Data":"6b70fc69f3ddea0f7ba61d03c2bebb161b89602633136d5f5252c6cf69d272d2"} Oct 30 21:39:41.537086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.537040 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:41.641366 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.641326 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:41.717154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.717125 4988 generic.go:356] "Generic (PLEG): container finished" podID="44562601-163d-400f-9394-8c5d6377eadb" containerID="7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458" exitCode=0 Oct 30 21:39:41.717323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.717184 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerDied","Data":"7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458"} Oct 30 21:39:41.738930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.738891 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:41.739090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.739060 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:41.791381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.791351 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:41.932082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:41.932039 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:42.002388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.002353 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:42.340734 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.340689 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:42.723695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.723663 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerStarted","Data":"59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab"} Oct 30 21:39:42.724103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.723728 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerStarted","Data":"680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b"} Oct 30 21:39:42.724103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.723920 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:42.847533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.847483 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=4.561704275 podStartE2EDuration="7.847468927s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285764652s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:42.846537377 +0000 UTC m=+3810.954033288" watchObservedRunningTime="2026-10-30 21:39:42.847468927 +0000 UTC m=+3810.954964873" Oct 30 21:39:42.897628 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:39:42.897582 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cc6051_3473_4d5a_ac8f_bb969dd32966.slice/crio-38e5f78b078fb8bc4388d54d59de34f3be04e10924934264f23362d45547ce54 WatchSource:0}: Error finding container 38e5f78b078fb8bc4388d54d59de34f3be04e10924934264f23362d45547ce54: Status 404 returned error can't find the container with id 38e5f78b078fb8bc4388d54d59de34f3be04e10924934264f23362d45547ce54 Oct 30 21:39:42.898397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:42.898374 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:39:43.730339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:43.730250 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerStarted","Data":"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9"} Oct 30 21:39:43.730339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:43.730316 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerStarted","Data":"38e5f78b078fb8bc4388d54d59de34f3be04e10924934264f23362d45547ce54"} Oct 30 21:39:47.749783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:47.749748 4988 generic.go:356] "Generic (PLEG): container finished" podID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerID="121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9" exitCode=0 Oct 30 21:39:47.750180 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:47.749819 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerDied","Data":"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9"} Oct 30 21:39:48.731672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:48.731634 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:39:48.756963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:48.756929 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerStarted","Data":"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d"} Oct 30 21:39:48.757349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:48.756995 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerStarted","Data":"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b"} Oct 30 21:39:48.757349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:48.757021 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerStarted","Data":"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775"} Oct 30 21:39:48.984315 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:48.984219 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.499512251 podStartE2EDuration="13.984202622s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484690371s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:48.982963349 +0000 UTC m=+3817.090459260" watchObservedRunningTime="2026-10-30 21:39:48.984202622 +0000 UTC m=+3817.091698533" Oct 30 21:39:52.340885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:52.340789 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:56.394453 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:39:56.394395 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:39:57.341450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:57.341389 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:57.347021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:57.346994 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:39:57.802073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:39:57.802043 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:13.896777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:13.896735 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.043124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:14.043094 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.244919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:14.244834 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.390701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:14.390667 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.535201 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:14.535115 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.679892 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:14.679858 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:18.876235 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:18.876198 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:40:18.876706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:18.876526 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="prometheus" containerID="cri-o://ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" gracePeriod=600 Oct 30 21:40:18.876706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:18.876539 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="config-reloader" containerID="cri-o://df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" gracePeriod=600 Oct 30 21:40:18.876706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:18.876553 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="thanos-sidecar" containerID="cri-o://c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" gracePeriod=600 Oct 30 21:40:19.638733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.638697 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:19.638947 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.638910 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="dbcb5b2a-395e-478b-818d-6321e760cba4" containerName="otc-container" containerID="cri-o://260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb" gracePeriod=30 Oct 30 21:40:19.669168 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.669139 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:19.790577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790478 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790525 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790548 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790663 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790701 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790733 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790789 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.790876 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790822 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.791131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790904 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.791131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.790973 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.791131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.791012 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.791131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.791042 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out\" (UniqueName: \"kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.791131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.791064 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:19.791131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.791094 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"75cc6051-3473-4d5a-ac8f-bb969dd32966\" (UID: \"75cc6051-3473-4d5a-ac8f-bb969dd32966\") " Oct 30 21:40:19.791461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.791356 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.791461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.791377 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:19.792325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.792049 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:19.793231 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.793194 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:19.793922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.793872 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.794034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.793958 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.794034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.793975 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "kube-api-access-68ccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:19.794875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.794843 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.794964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.794881 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.795031 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.794997 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.795119 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.795104 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:19.796627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.796610 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:40:19.806851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.806815 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config" pod "75cc6051-3473-4d5a-ac8f-bb969dd32966" (UID: "75cc6051-3473-4d5a-ac8f-bb969dd32966"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:19.892656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892614 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.892656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892655 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892671 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892686 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75cc6051-3473-4d5a-ac8f-bb969dd32966-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892701 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75cc6051-3473-4d5a-ac8f-bb969dd32966-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892718 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ccs\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-kube-api-access-68ccs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892734 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892749 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75cc6051-3473-4d5a-ac8f-bb969dd32966-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892841 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892856 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892880 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.893241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.892896 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/75cc6051-3473-4d5a-ac8f-bb969dd32966-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:19.904744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904710 4988 generic.go:356] "Generic (PLEG): container finished" podID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerID="c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" exitCode=0 Oct 30 21:40:19.904744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904746 4988 generic.go:356] "Generic (PLEG): container finished" podID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerID="df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" exitCode=0 Oct 30 21:40:19.904744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904756 4988 generic.go:356] "Generic (PLEG): container finished" podID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerID="ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" exitCode=0 Oct 30 21:40:19.905000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904795 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:19.905000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904793 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerDied","Data":"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d"} Oct 30 21:40:19.905000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904937 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerDied","Data":"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b"} Oct 30 21:40:19.905000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904975 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerDied","Data":"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775"} Oct 30 21:40:19.905000 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.904997 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"75cc6051-3473-4d5a-ac8f-bb969dd32966","Type":"ContainerDied","Data":"38e5f78b078fb8bc4388d54d59de34f3be04e10924934264f23362d45547ce54"} Oct 30 21:40:19.905244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.905024 4988 scope.go:122] "RemoveContainer" containerID="c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" Oct 30 21:40:19.909087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.909062 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:40:19.917314 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.917291 4988 scope.go:122] "RemoveContainer" containerID="df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" Oct 30 21:40:19.926400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.926376 4988 scope.go:122] "RemoveContainer" containerID="ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" Oct 30 21:40:19.935123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.935093 4988 scope.go:122] "RemoveContainer" containerID="121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9" Oct 30 21:40:19.944475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.944447 4988 scope.go:122] "RemoveContainer" containerID="c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" Oct 30 21:40:19.944834 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:19.944814 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": container with ID starting with c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d not found: ID does not exist" containerID="c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" Oct 30 21:40:19.944890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.944845 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d"} err="failed to get container status \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": rpc error: code = NotFound desc = could not find container \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": container with ID starting with c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d not found: ID does not exist" Oct 30 21:40:19.944890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.944863 4988 scope.go:122] "RemoveContainer" containerID="df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" Oct 30 21:40:19.945137 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:19.945113 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": container with ID starting with df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b not found: ID does not exist" containerID="df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" Oct 30 21:40:19.945205 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945145 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b"} err="failed to get container status \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": rpc error: code = NotFound desc = could not find container \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": container with ID starting with df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b not found: ID does not exist" Oct 30 21:40:19.945205 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945172 4988 scope.go:122] "RemoveContainer" containerID="ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" Oct 30 21:40:19.945435 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:19.945396 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": container with ID starting with ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775 not found: ID does not exist" containerID="ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" Oct 30 21:40:19.945491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945433 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775"} err="failed to get container status \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": rpc error: code = NotFound desc = could not find container \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": container with ID starting with ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775 not found: ID does not exist" Oct 30 21:40:19.945491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945448 4988 scope.go:122] "RemoveContainer" containerID="121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9" Oct 30 21:40:19.945684 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:19.945644 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": container with ID starting with 121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9 not found: ID does not exist" containerID="121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9" Oct 30 21:40:19.945762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945679 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9"} err="failed to get container status \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": rpc error: code = NotFound desc = could not find container \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": container with ID starting with 121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9 not found: ID does not exist" Oct 30 21:40:19.945762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945699 4988 scope.go:122] "RemoveContainer" containerID="c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" Oct 30 21:40:19.945916 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945896 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d"} err="failed to get container status \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": rpc error: code = NotFound desc = could not find container \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": container with ID starting with c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d not found: ID does not exist" Oct 30 21:40:19.945968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.945916 4988 scope.go:122] "RemoveContainer" containerID="df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" Oct 30 21:40:19.946142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946122 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b"} err="failed to get container status \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": rpc error: code = NotFound desc = could not find container \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": container with ID starting with df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b not found: ID does not exist" Oct 30 21:40:19.946212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946142 4988 scope.go:122] "RemoveContainer" containerID="ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" Oct 30 21:40:19.946346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946329 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775"} err="failed to get container status \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": rpc error: code = NotFound desc = could not find container \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": container with ID starting with ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775 not found: ID does not exist" Oct 30 21:40:19.946391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946347 4988 scope.go:122] "RemoveContainer" containerID="121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9" Oct 30 21:40:19.946582 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946565 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9"} err="failed to get container status \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": rpc error: code = NotFound desc = could not find container \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": container with ID starting with 121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9 not found: ID does not exist" Oct 30 21:40:19.946660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946583 4988 scope.go:122] "RemoveContainer" containerID="c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d" Oct 30 21:40:19.946799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946777 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d"} err="failed to get container status \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": rpc error: code = NotFound desc = could not find container \"c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d\": container with ID starting with c0f9177c9b57416364de7e709e455cadf6fda9df8d2f3eb399b0222a1297282d not found: ID does not exist" Oct 30 21:40:19.946865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946797 4988 scope.go:122] "RemoveContainer" containerID="df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b" Oct 30 21:40:19.947019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.946999 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b"} err="failed to get container status \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": rpc error: code = NotFound desc = could not find container \"df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b\": container with ID starting with df30991dfbf590aa03f5241116f051d72284496fa104ece30d1a719013af351b not found: ID does not exist" Oct 30 21:40:19.947065 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.947020 4988 scope.go:122] "RemoveContainer" containerID="ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775" Oct 30 21:40:19.947228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.947211 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775"} err="failed to get container status \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": rpc error: code = NotFound desc = could not find container \"ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775\": container with ID starting with ef8be49270a82ae5bc29ee3e67420ecad8b3e3832053fef66316fb8735ff0775 not found: ID does not exist" Oct 30 21:40:19.947296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.947228 4988 scope.go:122] "RemoveContainer" containerID="121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9" Oct 30 21:40:19.947457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.947436 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9"} err="failed to get container status \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": rpc error: code = NotFound desc = could not find container \"121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9\": container with ID starting with 121b86f7adb7e71f30d8933f4bf3654dedb7b41398c7aba92e649849fc8684c9 not found: ID does not exist" Oct 30 21:40:19.976469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.976439 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:19.993541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:19.993508 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:40:20.076426 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:20.076317 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:40:20.081826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:20.081797 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:40:20.129160 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:20.129124 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:20.330820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:20.330726 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:21.884015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.883984 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:21.915868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.915835 4988 generic.go:356] "Generic (PLEG): container finished" podID="dbcb5b2a-395e-478b-818d-6321e760cba4" containerID="260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb" exitCode=0 Oct 30 21:40:21.916032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.915885 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:21.916032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.915920 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dbcb5b2a-395e-478b-818d-6321e760cba4","Type":"ContainerDied","Data":"260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb"} Oct 30 21:40:21.916032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.915976 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dbcb5b2a-395e-478b-818d-6321e760cba4","Type":"ContainerDied","Data":"4c7095f887152758f518461d834076d603e2cf7202bb356767aaf13d70228ba2"} Oct 30 21:40:21.916032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.915999 4988 scope.go:122] "RemoveContainer" containerID="260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb" Oct 30 21:40:21.926640 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.926590 4988 scope.go:122] "RemoveContainer" containerID="260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb" Oct 30 21:40:21.927137 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:21.927113 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb\": container with ID starting with 260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb not found: ID does not exist" containerID="260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb" Oct 30 21:40:21.927259 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:21.927143 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb"} err="failed to get container status \"260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb\": rpc error: code = NotFound desc = could not find container \"260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb\": container with ID starting with 260ca71895ce7634274a2ffd81e2405b4940dc74a7318b31df3583c0ff6e20fb not found: ID does not exist" Oct 30 21:40:22.012482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012384 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal\") pod \"dbcb5b2a-395e-478b-818d-6321e760cba4\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " Oct 30 21:40:22.012482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012465 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj\") pod \"dbcb5b2a-395e-478b-818d-6321e760cba4\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " Oct 30 21:40:22.012723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012551 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs\") pod \"dbcb5b2a-395e-478b-818d-6321e760cba4\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " Oct 30 21:40:22.012723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012588 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs\") pod \"dbcb5b2a-395e-478b-818d-6321e760cba4\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " Oct 30 21:40:22.012723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012631 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert\") pod \"dbcb5b2a-395e-478b-818d-6321e760cba4\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " Oct 30 21:40:22.012723 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012660 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs\") pod \"dbcb5b2a-395e-478b-818d-6321e760cba4\" (UID: \"dbcb5b2a-395e-478b-818d-6321e760cba4\") " Oct 30 21:40:22.012965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.012942 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal" pod "dbcb5b2a-395e-478b-818d-6321e760cba4" (UID: "dbcb5b2a-395e-478b-818d-6321e760cba4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:22.015309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.015282 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs" pod "dbcb5b2a-395e-478b-818d-6321e760cba4" (UID: "dbcb5b2a-395e-478b-818d-6321e760cba4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:22.015481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.015400 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert" pod "dbcb5b2a-395e-478b-818d-6321e760cba4" (UID: "dbcb5b2a-395e-478b-818d-6321e760cba4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:22.016109 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.016041 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj" pod "dbcb5b2a-395e-478b-818d-6321e760cba4" (UID: "dbcb5b2a-395e-478b-818d-6321e760cba4"). InnerVolumeSpecName "kube-api-access-5cccj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:22.017546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.017518 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs" pod "dbcb5b2a-395e-478b-818d-6321e760cba4" (UID: "dbcb5b2a-395e-478b-818d-6321e760cba4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:22.018643 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.018616 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" path="/var/lib/kubelet/pods/75cc6051-3473-4d5a-ac8f-bb969dd32966/volumes" Oct 30 21:40:22.019268 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.019247 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs" pod "dbcb5b2a-395e-478b-818d-6321e760cba4" (UID: "dbcb5b2a-395e-478b-818d-6321e760cba4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:22.113530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.113482 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.113530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.113520 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.113530 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.113531 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.113786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.113542 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbcb5b2a-395e-478b-818d-6321e760cba4-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.113786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.113552 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dbcb5b2a-395e-478b-818d-6321e760cba4-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.113786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.113561 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cccj\" (UniqueName: \"kubernetes.io/projected/dbcb5b2a-395e-478b-818d-6321e760cba4-kube-api-access-5cccj\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.337999 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.337906 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:22.342638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.342611 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:22.419897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.419858 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:22.420280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420261 4988 cpu_manager.go:414] "containerMap: removing container" podUID="dbcb5b2a-395e-478b-818d-6321e760cba4" containerName="otc-container" Oct 30 21:40:22.420373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420282 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcb5b2a-395e-478b-818d-6321e760cba4" containerName="otc-container" Oct 30 21:40:22.420373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420299 4988 cpu_manager.go:414] "containerMap: removing container" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="config-reloader" Oct 30 21:40:22.420373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420306 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="config-reloader" Oct 30 21:40:22.420373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420334 4988 cpu_manager.go:414] "containerMap: removing container" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="thanos-sidecar" Oct 30 21:40:22.420373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420341 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="thanos-sidecar" Oct 30 21:40:22.420373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420370 4988 cpu_manager.go:414] "containerMap: removing container" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="prometheus" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420378 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="prometheus" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420392 4988 cpu_manager.go:414] "containerMap: removing container" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="init-config-reloader" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420400 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="init-config-reloader" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420509 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcb5b2a-395e-478b-818d-6321e760cba4" containerName="otc-container" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420524 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="config-reloader" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420536 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="thanos-sidecar" Oct 30 21:40:22.420712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.420549 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cc6051-3473-4d5a-ac8f-bb969dd32966" containerName="prometheus" Oct 30 21:40:22.424946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.424923 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.478066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.478027 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:40:22.478066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.478039 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-khslt\"" Oct 30 21:40:22.478066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.478066 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:40:22.478323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.478092 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:40:22.478323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.478164 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:40:22.478323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.478226 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:40:22.485508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.485474 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:22.517507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.517458 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.517712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.517586 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.517712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.517622 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.517712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.517642 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.517856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.517743 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk68\" (UniqueName: \"kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.517856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.517773 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.618965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.618866 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.618965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.618929 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.618965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.618948 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.618965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.618964 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.619311 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.618991 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk68\" (UniqueName: \"kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.619311 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.619027 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.619884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.619851 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.621548 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.621519 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.621676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.621632 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.621676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.621636 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.621784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.621704 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.680568 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.680534 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk68\" (UniqueName: \"kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68\") pod \"data-science-collector-collector-1\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:22.737330 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:22.737294 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:23.116164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:23.116127 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:23.119675 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:40:23.119640 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b8c5f0_ecb4_45ff_aab0_8b6d88ea781b.slice/crio-65dbce900af25ff5a788fb627338da4826da06e837659fa5a9c64d09fc42d6e9 WatchSource:0}: Error finding container 65dbce900af25ff5a788fb627338da4826da06e837659fa5a9c64d09fc42d6e9: Status 404 returned error can't find the container with id 65dbce900af25ff5a788fb627338da4826da06e837659fa5a9c64d09fc42d6e9 Oct 30 21:40:23.121774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:23.121739 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:23.121875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:23.121804 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:23.368811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:23.368722 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.929751 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:23.929714 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b","Type":"ContainerStarted","Data":"005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1"} Oct 30 21:40:23.929947 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:23.929772 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b","Type":"ContainerStarted","Data":"65dbce900af25ff5a788fb627338da4826da06e837659fa5a9c64d09fc42d6e9"} Oct 30 21:40:24.015474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:24.015443 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbcb5b2a-395e-478b-818d-6321e760cba4" path="/var/lib/kubelet/pods/dbcb5b2a-395e-478b-818d-6321e760cba4/volumes" Oct 30 21:40:24.049367 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:24.049307 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.049292516 podStartE2EDuration="2.049292516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:24.04800441 +0000 UTC m=+3852.155500322" watchObservedRunningTime="2026-10-30 21:40:24.049292516 +0000 UTC m=+3852.156788427" Oct 30 21:40:26.394369 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:26.394329 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:40:45.752238 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:45.752197 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:45.986970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:45.986931 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:46.198477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:46.198439 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:46.341266 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:46.341225 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:46.487489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:46.487375 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:46.634310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:46.634271 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:49.887258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.887220 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:40:49.890970 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.890953 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:49.944079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.944049 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:40:49.993470 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993432 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:40:49.993470 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993464 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:40:49.993470 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993445 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:40:49.993766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993473 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:40:49.993766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993486 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-jgwr8\"" Oct 30 21:40:49.993766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993438 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:40:49.993766 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.993584 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:40:49.995164 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:49.995146 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:40:50.044094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.044068 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:40:50.044364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.044344 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:40:50.159221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159132 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159221 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159190 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159262 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159303 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159326 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159443 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159468 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159531 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159557 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnvk\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159615 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159638 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.159709 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.159656 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.260778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260736 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.260778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260778 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260797 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260829 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260867 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260884 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260908 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260936 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260965 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.260993 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.261053 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.261081 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnvk\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.261713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.261682 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.262371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.262345 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.263375 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.263345 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.263838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.263727 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.264018 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.263991 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.264126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.264103 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.264309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.264285 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.264309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.264301 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.264471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.264313 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.264471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.264432 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.265504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.265486 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:50.322786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:50.322755 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnvk\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:51.505526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:51.505485 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:51.506066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:51.505740 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" containerName="otc-container" containerID="cri-o://005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1" gracePeriod=30 Oct 30 21:40:51.842494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:51.842377 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:51.992724 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:51.992684 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.203691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.203647 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.746450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.746394 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:52.885377 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885347 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs\") pod \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " Oct 30 21:40:52.885569 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885397 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert\") pod \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " Oct 30 21:40:52.885569 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885448 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68\" (UniqueName: \"kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68\") pod \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " Oct 30 21:40:52.885569 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885467 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs\") pod \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " Oct 30 21:40:52.885569 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885548 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal\") pod \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " Oct 30 21:40:52.885801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885580 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs\") pod \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\" (UID: \"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b\") " Oct 30 21:40:52.885964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.885931 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal" pod "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" (UID: "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:52.887763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.887728 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs" pod "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" (UID: "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:52.887763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.887750 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs" pod "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" (UID: "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:52.887943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.887777 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert" pod "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" (UID: "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:52.888019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.888001 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68" pod "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" (UID: "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b"). InnerVolumeSpecName "kube-api-access-dgk68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:52.888074 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.888058 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs" pod "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" (UID: "e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:52.987134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.987098 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.987134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.987130 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.987134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.987140 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.987134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.987150 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.987457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.987162 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgk68\" (UniqueName: \"kubernetes.io/projected/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-kube-api-access-dgk68\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:52.987457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:52.987171 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:53.059514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.059476 4988 generic.go:356] "Generic (PLEG): container finished" podID="e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" containerID="005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1" exitCode=0 Oct 30 21:40:53.059739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.059526 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.059739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.059551 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b","Type":"ContainerDied","Data":"005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1"} Oct 30 21:40:53.059739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.059608 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b","Type":"ContainerDied","Data":"65dbce900af25ff5a788fb627338da4826da06e837659fa5a9c64d09fc42d6e9"} Oct 30 21:40:53.059739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.059631 4988 scope.go:122] "RemoveContainer" containerID="005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1" Oct 30 21:40:53.068824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.068804 4988 scope.go:122] "RemoveContainer" containerID="005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1" Oct 30 21:40:53.069147 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:53.069115 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1\": container with ID starting with 005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1 not found: ID does not exist" containerID="005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1" Oct 30 21:40:53.069193 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.069158 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1"} err="failed to get container status \"005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1\": rpc error: code = NotFound desc = could not find container \"005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1\": container with ID starting with 005c1cf2cd0a89983798d7307af8a18e887d9f04560144656a9fe95e012b0db1 not found: ID does not exist" Oct 30 21:40:53.269827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.269741 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:53.273431 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.273395 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:53.372702 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.372669 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:53.373047 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.373035 4988 cpu_manager.go:414] "containerMap: removing container" podUID="e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" containerName="otc-container" Oct 30 21:40:53.373092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.373049 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" containerName="otc-container" Oct 30 21:40:53.373127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.373113 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" containerName="otc-container" Oct 30 21:40:53.377494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.377478 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.389324 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.389291 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.389495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.389337 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.389495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.389453 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.389597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.389510 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.389597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.389533 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.389690 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.389635 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.435664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.435632 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:40:53.435664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.435633 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:40:53.435884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.435633 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-khslt\"" Oct 30 21:40:53.435884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.435650 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:40:53.435884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.435688 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:40:53.438374 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.438346 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:40:53.440504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.440476 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:53.490399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.490357 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.490399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.490432 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.490710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.490451 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.490710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.490484 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.490710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.490513 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.490710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.490631 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.491210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.491185 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.492920 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.492877 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.493010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.492964 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.493218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.493202 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.493294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.493270 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.552536 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.552451 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv\") pod \"data-science-collector-collector-1\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.690220 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.690184 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:40:53.692527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.692502 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:53.797022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.796990 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:53.893851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.893817 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:53.894023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.893942 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:53.945778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:53.945735 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:54.015315 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.015281 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b" path="/var/lib/kubelet/pods/e7b8c5f0-ecb4-45ff-aab0-8b6d88ea781b/volumes" Oct 30 21:40:54.069306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.069268 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:40:54.072878 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:40:54.072848 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a73a047_1916_41d9_a828_9cd62ecc30d8.slice/crio-e07f787c686fbf8c4e114e860650ed0c5854830684060a2622c4fe966968a639 WatchSource:0}: Error finding container e07f787c686fbf8c4e114e860650ed0c5854830684060a2622c4fe966968a639: Status 404 returned error can't find the container with id e07f787c686fbf8c4e114e860650ed0c5854830684060a2622c4fe966968a639 Oct 30 21:40:54.075447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.075381 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:54.075545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.075489 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:54.090424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.090379 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:54.157620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.157532 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:54.410574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.410479 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:40:54.895756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:54.895715 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:40:54.950195 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:40:54.950154 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422b8946_a54d_47a5_8de7_66c7d6953723.slice/crio-c544e7b4c3629b6f460aa4a9044519ff2e57021e09078ea8c68bd5ee4135fc05 WatchSource:0}: Error finding container c544e7b4c3629b6f460aa4a9044519ff2e57021e09078ea8c68bd5ee4135fc05: Status 404 returned error can't find the container with id c544e7b4c3629b6f460aa4a9044519ff2e57021e09078ea8c68bd5ee4135fc05 Oct 30 21:40:55.071264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:55.071222 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerStarted","Data":"c544e7b4c3629b6f460aa4a9044519ff2e57021e09078ea8c68bd5ee4135fc05"} Oct 30 21:40:55.072977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:55.072951 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2a73a047-1916-41d9-a828-9cd62ecc30d8","Type":"ContainerStarted","Data":"c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f"} Oct 30 21:40:55.073117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:55.072995 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2a73a047-1916-41d9-a828-9cd62ecc30d8","Type":"ContainerStarted","Data":"e07f787c686fbf8c4e114e860650ed0c5854830684060a2622c4fe966968a639"} Oct 30 21:40:55.192005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:55.191945 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.191932677 podStartE2EDuration="2.191932677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:55.190555857 +0000 UTC m=+3883.298051779" watchObservedRunningTime="2026-10-30 21:40:55.191932677 +0000 UTC m=+3883.299428587" Oct 30 21:40:56.080378 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:56.080340 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerStarted","Data":"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83"} Oct 30 21:40:56.394730 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:40:56.394689 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:40:59.096202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:59.096122 4988 generic.go:356] "Generic (PLEG): container finished" podID="422b8946-a54d-47a5-8de7-66c7d6953723" containerID="6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83" exitCode=0 Oct 30 21:40:59.096202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:40:59.096173 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerDied","Data":"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83"} Oct 30 21:41:00.104463 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:00.104397 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerStarted","Data":"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2"} Oct 30 21:41:00.105024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:00.104508 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerStarted","Data":"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b"} Oct 30 21:41:00.105024 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:00.104546 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerStarted","Data":"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba"} Oct 30 21:41:00.281887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:00.281833 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=7.995366876 podStartE2EDuration="11.281820448s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286453572s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:00.280627744 +0000 UTC m=+3888.388123653" watchObservedRunningTime="2026-10-30 21:41:00.281820448 +0000 UTC m=+3888.389316358" Oct 30 21:41:04.410819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:04.410780 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:41:09.411504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:09.411466 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:41:09.416868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:09.416843 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:41:10.156714 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:10.156684 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:41:12.396166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:12.396137 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:41:12.411457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:12.404005 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:41:23.718676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:23.718586 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:26.393570 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:26.393534 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:41:28.965767 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:28.965731 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.113076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:29.113027 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.314804 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:29.314707 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.459910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:29.459872 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.605679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:29.605588 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.752764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:29.752725 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:29.899148 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:29.899111 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:34.744624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:34.744584 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:34.745068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:34.744785 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2a73a047-1916-41d9-a828-9cd62ecc30d8" containerName="otc-container" containerID="cri-o://c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f" gracePeriod=30 Oct 30 21:41:35.081142 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:35.081050 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:35.236204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:35.236167 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:35.434998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:35.434963 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:37.099556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.099528 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.278398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278311 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs\") pod \"2a73a047-1916-41d9-a828-9cd62ecc30d8\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " Oct 30 21:41:37.278398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278330 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2a73a047-1916-41d9-a828-9cd62ecc30d8","Type":"ContainerDied","Data":"c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f"} Oct 30 21:41:37.278398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278349 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.278398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278388 4988 scope.go:122] "RemoveContainer" containerID="c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f" Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278385 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert\") pod \"2a73a047-1916-41d9-a828-9cd62ecc30d8\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278305 4988 generic.go:356] "Generic (PLEG): container finished" podID="2a73a047-1916-41d9-a828-9cd62ecc30d8" containerID="c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f" exitCode=0 Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278594 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs\") pod \"2a73a047-1916-41d9-a828-9cd62ecc30d8\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278627 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal\") pod \"2a73a047-1916-41d9-a828-9cd62ecc30d8\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278628 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2a73a047-1916-41d9-a828-9cd62ecc30d8","Type":"ContainerDied","Data":"e07f787c686fbf8c4e114e860650ed0c5854830684060a2622c4fe966968a639"} Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278646 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv\") pod \"2a73a047-1916-41d9-a828-9cd62ecc30d8\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " Oct 30 21:41:37.278759 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.278706 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs\") pod \"2a73a047-1916-41d9-a828-9cd62ecc30d8\" (UID: \"2a73a047-1916-41d9-a828-9cd62ecc30d8\") " Oct 30 21:41:37.279219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.279192 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal" pod "2a73a047-1916-41d9-a828-9cd62ecc30d8" (UID: "2a73a047-1916-41d9-a828-9cd62ecc30d8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:37.280883 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.280852 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert" pod "2a73a047-1916-41d9-a828-9cd62ecc30d8" (UID: "2a73a047-1916-41d9-a828-9cd62ecc30d8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:37.281056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.281026 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv" pod "2a73a047-1916-41d9-a828-9cd62ecc30d8" (UID: "2a73a047-1916-41d9-a828-9cd62ecc30d8"). InnerVolumeSpecName "kube-api-access-w8spv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:37.281147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.281060 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs" pod "2a73a047-1916-41d9-a828-9cd62ecc30d8" (UID: "2a73a047-1916-41d9-a828-9cd62ecc30d8"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:37.281319 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.281303 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs" pod "2a73a047-1916-41d9-a828-9cd62ecc30d8" (UID: "2a73a047-1916-41d9-a828-9cd62ecc30d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:37.281468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.281374 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs" pod "2a73a047-1916-41d9-a828-9cd62ecc30d8" (UID: "2a73a047-1916-41d9-a828-9cd62ecc30d8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:37.302098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.302072 4988 scope.go:122] "RemoveContainer" containerID="c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f" Oct 30 21:41:37.302488 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:37.302463 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f\": container with ID starting with c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f not found: ID does not exist" containerID="c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f" Oct 30 21:41:37.302556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.302502 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f"} err="failed to get container status \"c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f\": rpc error: code = NotFound desc = could not find container \"c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f\": container with ID starting with c1d31ed00803d8d2cfcf29b62be3ca9bea26e36ad47ecc34ba03b6020c85250f not found: ID does not exist" Oct 30 21:41:37.379353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.379318 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:37.379353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.379348 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:37.379353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.379359 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2a73a047-1916-41d9-a828-9cd62ecc30d8-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:37.379632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.379368 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8spv\" (UniqueName: \"kubernetes.io/projected/2a73a047-1916-41d9-a828-9cd62ecc30d8-kube-api-access-w8spv\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:37.379632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.379379 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:37.379632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.379387 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a73a047-1916-41d9-a828-9cd62ecc30d8-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:37.699339 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.699298 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:37.705501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.705470 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:37.792903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.792866 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:37.793275 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.793260 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2a73a047-1916-41d9-a828-9cd62ecc30d8" containerName="otc-container" Oct 30 21:41:37.793319 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.793277 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a73a047-1916-41d9-a828-9cd62ecc30d8" containerName="otc-container" Oct 30 21:41:37.793375 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.793367 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a73a047-1916-41d9-a828-9cd62ecc30d8" containerName="otc-container" Oct 30 21:41:37.797548 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.797526 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.849910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.849811 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:41:37.854997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.854967 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:37.892029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.891991 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:41:37.892029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.892003 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-khslt\"" Oct 30 21:41:37.892287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.892052 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:41:37.892287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.892112 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:41:37.892287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.892190 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:41:37.984783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.984746 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.984783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.984788 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.985019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.984819 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.985019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.984946 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.985019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.984974 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:37.985121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:37.985104 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.015424 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.015380 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a73a047-1916-41d9-a828-9cd62ecc30d8" path="/var/lib/kubelet/pods/2a73a047-1916-41d9-a828-9cd62ecc30d8/volumes" Oct 30 21:41:38.085770 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.085732 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.085770 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.085773 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.086038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.085819 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.086038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.085839 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.086038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.085858 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.086038 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.085950 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.086706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.086666 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.088633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.088600 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.088750 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.088601 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.088931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.088907 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.089012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.088994 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.147277 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.147235 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9\") pod \"data-science-collector-collector-1\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.410622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.410537 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:38.786742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.786657 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:38.818404 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:41:38.818369 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b48d5cc_0671_4372_ad86_6eb593c320ad.slice/crio-0f30f528b10237866851320bc6878cb2f1ef3e9281e8d634d3f75588e0121d72 WatchSource:0}: Error finding container 0f30f528b10237866851320bc6878cb2f1ef3e9281e8d634d3f75588e0121d72: Status 404 returned error can't find the container with id 0f30f528b10237866851320bc6878cb2f1ef3e9281e8d634d3f75588e0121d72 Oct 30 21:41:38.820585 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.820548 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:38.820683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:38.820617 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:41:39.291762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:39.291721 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b48d5cc-0671-4372-ad86-6eb593c320ad","Type":"ContainerStarted","Data":"84aa4d55b9b128c50711a9676a6832dd22f1fe7d2ff57667011d83c31553b791"} Oct 30 21:41:39.292209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:39.291782 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b48d5cc-0671-4372-ad86-6eb593c320ad","Type":"ContainerStarted","Data":"0f30f528b10237866851320bc6878cb2f1ef3e9281e8d634d3f75588e0121d72"} Oct 30 21:41:39.410365 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:39.410306 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.410290478 podStartE2EDuration="2.410290478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:39.409492478 +0000 UTC m=+3927.516988388" watchObservedRunningTime="2026-10-30 21:41:39.410290478 +0000 UTC m=+3927.517786389" Oct 30 21:41:52.559666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.559623 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7"] Oct 30 21:41:52.560189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.559973 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="kube-rbac-proxy" containerID="cri-o://13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" gracePeriod=30 Oct 30 21:41:52.560189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.560063 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="method-gate" containerID="cri-o://d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" gracePeriod=30 Oct 30 21:41:52.560189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.560063 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="prom-label-proxy" containerID="cri-o://6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" gracePeriod=30 Oct 30 21:41:52.663340 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.663296 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc"] Oct 30 21:41:52.663655 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.663622 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="kube-rbac-proxy" containerID="cri-o://cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" gracePeriod=30 Oct 30 21:41:52.663782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.663626 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="prom-label-proxy" containerID="cri-o://6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" gracePeriod=30 Oct 30 21:41:52.663825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.663632 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="method-gate" containerID="cri-o://3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" gracePeriod=30 Oct 30 21:41:52.825772 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.825744 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:41:52.902520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.902488 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca\") pod \"53bb217e-6b85-4abf-8588-db4652cabe16\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " Oct 30 21:41:52.902732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.902537 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config\") pod \"53bb217e-6b85-4abf-8588-db4652cabe16\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " Oct 30 21:41:52.902732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.902563 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls\" (UniqueName: \"kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls\") pod \"53bb217e-6b85-4abf-8588-db4652cabe16\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " Oct 30 21:41:52.902921 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.902894 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca" pod "53bb217e-6b85-4abf-8588-db4652cabe16" (UID: "53bb217e-6b85-4abf-8588-db4652cabe16"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:52.903036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.903010 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config" pod "53bb217e-6b85-4abf-8588-db4652cabe16" (UID: "53bb217e-6b85-4abf-8588-db4652cabe16"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:52.905777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.905742 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls" pod "53bb217e-6b85-4abf-8588-db4652cabe16" (UID: "53bb217e-6b85-4abf-8588-db4652cabe16"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:52.929829 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:52.929801 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:41:53.003790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.003754 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config\" (UniqueName: \"kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config\") pod \"08701e18-d0af-45b3-a571-274982048103\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " Oct 30 21:41:53.003996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.003821 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca\") pod \"53bb217e-6b85-4abf-8588-db4652cabe16\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " Oct 30 21:41:53.003996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.003855 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm\" (UniqueName: \"kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm\") pod \"53bb217e-6b85-4abf-8588-db4652cabe16\" (UID: \"53bb217e-6b85-4abf-8588-db4652cabe16\") " Oct 30 21:41:53.003996 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.003913 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls\" (UniqueName: \"kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls\") pod \"08701e18-d0af-45b3-a571-274982048103\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " Oct 30 21:41:53.004178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.004160 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.004239 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.004182 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53bb217e-6b85-4abf-8588-db4652cabe16-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.004239 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.004179 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config" pod "08701e18-d0af-45b3-a571-274982048103" (UID: "08701e18-d0af-45b3-a571-274982048103"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:53.004239 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.004197 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.004239 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.004223 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca" pod "53bb217e-6b85-4abf-8588-db4652cabe16" (UID: "53bb217e-6b85-4abf-8588-db4652cabe16"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:53.006079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.006050 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm" pod "53bb217e-6b85-4abf-8588-db4652cabe16" (UID: "53bb217e-6b85-4abf-8588-db4652cabe16"). InnerVolumeSpecName "kube-api-access-zs9tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:53.006203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.006094 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls" pod "08701e18-d0af-45b3-a571-274982048103" (UID: "08701e18-d0af-45b3-a571-274982048103"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:53.104633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.104542 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp\" (UniqueName: \"kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp\") pod \"08701e18-d0af-45b3-a571-274982048103\" (UID: \"08701e18-d0af-45b3-a571-274982048103\") " Oct 30 21:41:53.104814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.104769 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/08701e18-d0af-45b3-a571-274982048103-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.104814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.104787 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08701e18-d0af-45b3-a571-274982048103-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.104814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.104799 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/53bb217e-6b85-4abf-8588-db4652cabe16-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.104814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.104811 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs9tm\" (UniqueName: \"kubernetes.io/projected/53bb217e-6b85-4abf-8588-db4652cabe16-kube-api-access-zs9tm\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.106952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.106919 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp" pod "08701e18-d0af-45b3-a571-274982048103" (UID: "08701e18-d0af-45b3-a571-274982048103"). InnerVolumeSpecName "kube-api-access-kj2dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:53.205196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.205153 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2dp\" (UniqueName: \"kubernetes.io/projected/08701e18-d0af-45b3-a571-274982048103-kube-api-access-kj2dp\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:53.358664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358573 4988 generic.go:356] "Generic (PLEG): container finished" podID="53bb217e-6b85-4abf-8588-db4652cabe16" containerID="6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" exitCode=0 Oct 30 21:41:53.358664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358603 4988 generic.go:356] "Generic (PLEG): container finished" podID="53bb217e-6b85-4abf-8588-db4652cabe16" containerID="d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" exitCode=0 Oct 30 21:41:53.358664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358617 4988 generic.go:356] "Generic (PLEG): container finished" podID="53bb217e-6b85-4abf-8588-db4652cabe16" containerID="13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" exitCode=0 Oct 30 21:41:53.358664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358630 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" Oct 30 21:41:53.358664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358659 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerDied","Data":"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b"} Oct 30 21:41:53.359058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358724 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerDied","Data":"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276"} Oct 30 21:41:53.359058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358742 4988 scope.go:122] "RemoveContainer" containerID="6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" Oct 30 21:41:53.359058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358749 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerDied","Data":"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d"} Oct 30 21:41:53.359058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.358877 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7" event={"ID":"53bb217e-6b85-4abf-8588-db4652cabe16","Type":"ContainerDied","Data":"8edaa9a363fbc96535f7aca1cd7bb41240a8c15b0b07492942be04091aa871dc"} Oct 30 21:41:53.360896 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.360862 4988 generic.go:356] "Generic (PLEG): container finished" podID="08701e18-d0af-45b3-a571-274982048103" containerID="6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" exitCode=0 Oct 30 21:41:53.360896 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.360894 4988 generic.go:356] "Generic (PLEG): container finished" podID="08701e18-d0af-45b3-a571-274982048103" containerID="3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" exitCode=0 Oct 30 21:41:53.361050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.360904 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" Oct 30 21:41:53.361050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.360922 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerDied","Data":"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3"} Oct 30 21:41:53.361050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.360986 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerDied","Data":"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7"} Oct 30 21:41:53.361050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.361023 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerDied","Data":"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db"} Oct 30 21:41:53.361050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.360906 4988 generic.go:356] "Generic (PLEG): container finished" podID="08701e18-d0af-45b3-a571-274982048103" containerID="cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" exitCode=0 Oct 30 21:41:53.361244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.361062 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc" event={"ID":"08701e18-d0af-45b3-a571-274982048103","Type":"ContainerDied","Data":"fe99a54659616d899ef56f39d6a4a78624cc1ff701b41573ccdd6f7f9dbf6f04"} Oct 30 21:41:53.368392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.368368 4988 scope.go:122] "RemoveContainer" containerID="d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" Oct 30 21:41:53.376780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.376752 4988 scope.go:122] "RemoveContainer" containerID="13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" Oct 30 21:41:53.385273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.385252 4988 scope.go:122] "RemoveContainer" containerID="6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" Oct 30 21:41:53.385636 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:53.385615 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": container with ID starting with 6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b not found: ID does not exist" containerID="6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" Oct 30 21:41:53.385702 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.385648 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b"} err="failed to get container status \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": rpc error: code = NotFound desc = could not find container \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": container with ID starting with 6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b not found: ID does not exist" Oct 30 21:41:53.385702 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.385666 4988 scope.go:122] "RemoveContainer" containerID="d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" Oct 30 21:41:53.385953 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:53.385933 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": container with ID starting with d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276 not found: ID does not exist" containerID="d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" Oct 30 21:41:53.386026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.385963 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276"} err="failed to get container status \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": rpc error: code = NotFound desc = could not find container \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": container with ID starting with d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276 not found: ID does not exist" Oct 30 21:41:53.386026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.385987 4988 scope.go:122] "RemoveContainer" containerID="13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" Oct 30 21:41:53.386242 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:53.386224 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": container with ID starting with 13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d not found: ID does not exist" containerID="13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" Oct 30 21:41:53.386278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386248 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d"} err="failed to get container status \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": rpc error: code = NotFound desc = could not find container \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": container with ID starting with 13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d not found: ID does not exist" Oct 30 21:41:53.386278 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386261 4988 scope.go:122] "RemoveContainer" containerID="6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" Oct 30 21:41:53.386515 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386493 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b"} err="failed to get container status \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": rpc error: code = NotFound desc = could not find container \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": container with ID starting with 6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b not found: ID does not exist" Oct 30 21:41:53.386580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386515 4988 scope.go:122] "RemoveContainer" containerID="d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" Oct 30 21:41:53.386755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386734 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276"} err="failed to get container status \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": rpc error: code = NotFound desc = could not find container \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": container with ID starting with d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276 not found: ID does not exist" Oct 30 21:41:53.386802 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386756 4988 scope.go:122] "RemoveContainer" containerID="13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" Oct 30 21:41:53.386969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386952 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d"} err="failed to get container status \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": rpc error: code = NotFound desc = could not find container \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": container with ID starting with 13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d not found: ID does not exist" Oct 30 21:41:53.387008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.386970 4988 scope.go:122] "RemoveContainer" containerID="6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b" Oct 30 21:41:53.387191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.387170 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b"} err="failed to get container status \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": rpc error: code = NotFound desc = could not find container \"6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b\": container with ID starting with 6b0ce246e4e912bc88a3980fc4323ac01055c866dde70c39b8a5c8bd50d0f18b not found: ID does not exist" Oct 30 21:41:53.387270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.387192 4988 scope.go:122] "RemoveContainer" containerID="d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276" Oct 30 21:41:53.387483 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.387466 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276"} err="failed to get container status \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": rpc error: code = NotFound desc = could not find container \"d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276\": container with ID starting with d1ee28db2adbf99158d112e3e45f309a8b6985ad6c10dcbd503369df2f183276 not found: ID does not exist" Oct 30 21:41:53.387532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.387483 4988 scope.go:122] "RemoveContainer" containerID="13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d" Oct 30 21:41:53.387738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.387719 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d"} err="failed to get container status \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": rpc error: code = NotFound desc = could not find container \"13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d\": container with ID starting with 13b95f693b466918b2982467e95b46606ef8796e25f37b860c1818edd2bedd3d not found: ID does not exist" Oct 30 21:41:53.387810 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.387738 4988 scope.go:122] "RemoveContainer" containerID="6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" Oct 30 21:41:53.396293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.396269 4988 scope.go:122] "RemoveContainer" containerID="3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" Oct 30 21:41:53.405107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.405083 4988 scope.go:122] "RemoveContainer" containerID="cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" Oct 30 21:41:53.413851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.413828 4988 scope.go:122] "RemoveContainer" containerID="6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" Oct 30 21:41:53.414228 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:53.414197 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": container with ID starting with 6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3 not found: ID does not exist" containerID="6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" Oct 30 21:41:53.414344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.414240 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3"} err="failed to get container status \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": rpc error: code = NotFound desc = could not find container \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": container with ID starting with 6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3 not found: ID does not exist" Oct 30 21:41:53.414344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.414264 4988 scope.go:122] "RemoveContainer" containerID="3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" Oct 30 21:41:53.414664 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:53.414641 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": container with ID starting with 3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7 not found: ID does not exist" containerID="3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" Oct 30 21:41:53.414732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.414673 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7"} err="failed to get container status \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": rpc error: code = NotFound desc = could not find container \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": container with ID starting with 3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7 not found: ID does not exist" Oct 30 21:41:53.414732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.414694 4988 scope.go:122] "RemoveContainer" containerID="cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" Oct 30 21:41:53.415050 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:53.415023 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": container with ID starting with cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db not found: ID does not exist" containerID="cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" Oct 30 21:41:53.415159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.415054 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db"} err="failed to get container status \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": rpc error: code = NotFound desc = could not find container \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": container with ID starting with cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db not found: ID does not exist" Oct 30 21:41:53.415159 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.415070 4988 scope.go:122] "RemoveContainer" containerID="6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" Oct 30 21:41:53.415793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.415764 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3"} err="failed to get container status \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": rpc error: code = NotFound desc = could not find container \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": container with ID starting with 6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3 not found: ID does not exist" Oct 30 21:41:53.415793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.415792 4988 scope.go:122] "RemoveContainer" containerID="3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" Oct 30 21:41:53.416123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.416082 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7"} err="failed to get container status \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": rpc error: code = NotFound desc = could not find container \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": container with ID starting with 3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7 not found: ID does not exist" Oct 30 21:41:53.416123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.416123 4988 scope.go:122] "RemoveContainer" containerID="cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" Oct 30 21:41:53.416467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.416445 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db"} err="failed to get container status \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": rpc error: code = NotFound desc = could not find container \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": container with ID starting with cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db not found: ID does not exist" Oct 30 21:41:53.416467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.416466 4988 scope.go:122] "RemoveContainer" containerID="6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3" Oct 30 21:41:53.416786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.416760 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3"} err="failed to get container status \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": rpc error: code = NotFound desc = could not find container \"6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3\": container with ID starting with 6d889ad610de4dca98e33fa8934916792fe5ce05ae408983f64790c3b0d5e8b3 not found: ID does not exist" Oct 30 21:41:53.416849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.416787 4988 scope.go:122] "RemoveContainer" containerID="3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7" Oct 30 21:41:53.417081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.417057 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7"} err="failed to get container status \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": rpc error: code = NotFound desc = could not find container \"3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7\": container with ID starting with 3f0f192e97abda7cb9f15ac30d9be671c6df213a48cb2c64dcaffa7dff6f09f7 not found: ID does not exist" Oct 30 21:41:53.417146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.417082 4988 scope.go:122] "RemoveContainer" containerID="cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db" Oct 30 21:41:53.417325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.417308 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db"} err="failed to get container status \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": rpc error: code = NotFound desc = could not find container \"cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db\": container with ID starting with cadde510291d63854072bf724aceaeb58a996219bb3bbf002603b8b1d396f0db not found: ID does not exist" Oct 30 21:41:53.525305 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.525262 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc"] Oct 30 21:41:53.529035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.529005 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-7srbc"] Oct 30 21:41:53.646274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.646237 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7"] Oct 30 21:41:53.655287 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.654104 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-x7nn7"] Oct 30 21:41:53.655479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.655296 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:53.901001 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.900906 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:53.937036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.936995 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:41:53.937312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.937277 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="prometheus" containerID="cri-o://4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" gracePeriod=600 Oct 30 21:41:53.937506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.937316 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="thanos-sidecar" containerID="cri-o://5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" gracePeriod=600 Oct 30 21:41:53.937506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.937318 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="config-reloader" containerID="cri-o://e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" gracePeriod=600 Oct 30 21:41:53.953323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.953287 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:41:53.953646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.953614 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="alertmanager" containerID="cri-o://680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b" gracePeriod=120 Oct 30 21:41:53.953718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:53.953677 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="config-reloader" containerID="cri-o://59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab" gracePeriod=120 Oct 30 21:41:54.015346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.015311 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08701e18-d0af-45b3-a571-274982048103" path="/var/lib/kubelet/pods/08701e18-d0af-45b3-a571-274982048103/volumes" Oct 30 21:41:54.015865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.015846 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" path="/var/lib/kubelet/pods/53bb217e-6b85-4abf-8588-db4652cabe16/volumes" Oct 30 21:41:54.107805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.107766 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:54.266120 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.266090 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:41:54.312906 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.312872 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313002 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313023 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313053 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313082 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313076 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313114 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313154 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313190 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313227 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313269 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313287 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:54.313322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313299 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313356 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313509 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out\" (UniqueName: \"kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out\") pod \"422b8946-a54d-47a5-8de7-66c7d6953723\" (UID: \"422b8946-a54d-47a5-8de7-66c7d6953723\") " Oct 30 21:41:54.313703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313533 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:54.313703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313593 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:54.313908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313760 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.313908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313780 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.313908 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.313795 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422b8946-a54d-47a5-8de7-66c7d6953723-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.315846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.315809 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "kube-api-access-bvnvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:54.315846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.315824 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.316075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.316043 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.316191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.316139 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:54.316261 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.316245 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.317549 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.317524 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:54.318105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.318071 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.318105 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.318100 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.318738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.318715 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:41:54.327676 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.327648 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:54.328760 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.328739 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config" pod "422b8946-a54d-47a5-8de7-66c7d6953723" (UID: "422b8946-a54d-47a5-8de7-66c7d6953723"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.367742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.367707 4988 generic.go:356] "Generic (PLEG): container finished" podID="44562601-163d-400f-9394-8c5d6377eadb" containerID="59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab" exitCode=0 Oct 30 21:41:54.367922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.367787 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerDied","Data":"59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab"} Oct 30 21:41:54.371645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371612 4988 generic.go:356] "Generic (PLEG): container finished" podID="422b8946-a54d-47a5-8de7-66c7d6953723" containerID="5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" exitCode=0 Oct 30 21:41:54.371645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371641 4988 generic.go:356] "Generic (PLEG): container finished" podID="422b8946-a54d-47a5-8de7-66c7d6953723" containerID="e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" exitCode=0 Oct 30 21:41:54.371645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371648 4988 generic.go:356] "Generic (PLEG): container finished" podID="422b8946-a54d-47a5-8de7-66c7d6953723" containerID="4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" exitCode=0 Oct 30 21:41:54.371855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371677 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:41:54.371855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371691 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerDied","Data":"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2"} Oct 30 21:41:54.371855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371756 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerDied","Data":"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b"} Oct 30 21:41:54.371855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371781 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerDied","Data":"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba"} Oct 30 21:41:54.371855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371803 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"422b8946-a54d-47a5-8de7-66c7d6953723","Type":"ContainerDied","Data":"c544e7b4c3629b6f460aa4a9044519ff2e57021e09078ea8c68bd5ee4135fc05"} Oct 30 21:41:54.371855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.371833 4988 scope.go:122] "RemoveContainer" containerID="5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" Oct 30 21:41:54.380609 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.380593 4988 scope.go:122] "RemoveContainer" containerID="e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" Oct 30 21:41:54.389482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.389459 4988 scope.go:122] "RemoveContainer" containerID="4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" Oct 30 21:41:54.398534 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.398507 4988 scope.go:122] "RemoveContainer" containerID="6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83" Oct 30 21:41:54.408942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.408917 4988 scope.go:122] "RemoveContainer" containerID="5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" Oct 30 21:41:54.409377 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:54.409343 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": container with ID starting with 5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2 not found: ID does not exist" containerID="5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" Oct 30 21:41:54.409500 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.409386 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2"} err="failed to get container status \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": rpc error: code = NotFound desc = could not find container \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": container with ID starting with 5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2 not found: ID does not exist" Oct 30 21:41:54.409500 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.409424 4988 scope.go:122] "RemoveContainer" containerID="e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" Oct 30 21:41:54.409766 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:54.409743 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": container with ID starting with e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b not found: ID does not exist" containerID="e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" Oct 30 21:41:54.409840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.409773 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b"} err="failed to get container status \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": rpc error: code = NotFound desc = could not find container \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": container with ID starting with e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b not found: ID does not exist" Oct 30 21:41:54.409840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.409793 4988 scope.go:122] "RemoveContainer" containerID="4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" Oct 30 21:41:54.410088 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:54.410067 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": container with ID starting with 4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba not found: ID does not exist" containerID="4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" Oct 30 21:41:54.410150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410099 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba"} err="failed to get container status \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": rpc error: code = NotFound desc = could not find container \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": container with ID starting with 4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba not found: ID does not exist" Oct 30 21:41:54.410150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410114 4988 scope.go:122] "RemoveContainer" containerID="6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83" Oct 30 21:41:54.410394 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:54.410377 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": container with ID starting with 6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83 not found: ID does not exist" containerID="6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83" Oct 30 21:41:54.410504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410398 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83"} err="failed to get container status \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": rpc error: code = NotFound desc = could not find container \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": container with ID starting with 6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83 not found: ID does not exist" Oct 30 21:41:54.410504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410445 4988 scope.go:122] "RemoveContainer" containerID="5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" Oct 30 21:41:54.410740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410715 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2"} err="failed to get container status \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": rpc error: code = NotFound desc = could not find container \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": container with ID starting with 5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2 not found: ID does not exist" Oct 30 21:41:54.410809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410742 4988 scope.go:122] "RemoveContainer" containerID="e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" Oct 30 21:41:54.411015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.410991 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b"} err="failed to get container status \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": rpc error: code = NotFound desc = could not find container \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": container with ID starting with e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b not found: ID does not exist" Oct 30 21:41:54.411078 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411015 4988 scope.go:122] "RemoveContainer" containerID="4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" Oct 30 21:41:54.411297 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411274 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba"} err="failed to get container status \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": rpc error: code = NotFound desc = could not find container \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": container with ID starting with 4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba not found: ID does not exist" Oct 30 21:41:54.411297 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411296 4988 scope.go:122] "RemoveContainer" containerID="6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83" Oct 30 21:41:54.411608 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411589 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83"} err="failed to get container status \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": rpc error: code = NotFound desc = could not find container \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": container with ID starting with 6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83 not found: ID does not exist" Oct 30 21:41:54.411608 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411607 4988 scope.go:122] "RemoveContainer" containerID="5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2" Oct 30 21:41:54.411845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411821 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2"} err="failed to get container status \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": rpc error: code = NotFound desc = could not find container \"5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2\": container with ID starting with 5bc87ae5e0bde19f63f0d537bb7006d45bc48c3135781b2ab386e07afcd3edd2 not found: ID does not exist" Oct 30 21:41:54.411925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.411848 4988 scope.go:122] "RemoveContainer" containerID="e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b" Oct 30 21:41:54.412136 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.412113 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b"} err="failed to get container status \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": rpc error: code = NotFound desc = could not find container \"e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b\": container with ID starting with e715354e24ea5263f1dce1de4664fa2fdc89ec2d4941425eb1ae428107e2db0b not found: ID does not exist" Oct 30 21:41:54.412189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.412137 4988 scope.go:122] "RemoveContainer" containerID="4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba" Oct 30 21:41:54.412390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.412365 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba"} err="failed to get container status \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": rpc error: code = NotFound desc = could not find container \"4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba\": container with ID starting with 4fe63e90bfccd35cc00d24e4530df1f36926fd49e7c38346454cd157e66769ba not found: ID does not exist" Oct 30 21:41:54.412468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.412392 4988 scope.go:122] "RemoveContainer" containerID="6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83" Oct 30 21:41:54.412650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.412631 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83"} err="failed to get container status \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": rpc error: code = NotFound desc = could not find container \"6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83\": container with ID starting with 6f644a97a36b9849e734c0f3654acf453d3ba98c291e516d2f0d50b197905e83 not found: ID does not exist" Oct 30 21:41:54.415206 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415178 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415215 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415234 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415250 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415266 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnvk\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-kube-api-access-bvnvk\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415280 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422b8946-a54d-47a5-8de7-66c7d6953723-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415293 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422b8946-a54d-47a5-8de7-66c7d6953723-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415332 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:41:54.415421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415347 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.415421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.415360 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/422b8946-a54d-47a5-8de7-66c7d6953723-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.430494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.430472 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:41:54.485439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.485349 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:54.512258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.512226 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:41:54.516352 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.516289 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:41:54.517359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.517335 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:41:54.809505 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.809475 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:41:54.818620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818584 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.818840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818635 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.818840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818669 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.818840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818700 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.818840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818750 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.818840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818788 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.819147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.818841 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume\") pod \"44562601-163d-400f-9394-8c5d6377eadb\" (UID: \"44562601-163d-400f-9394-8c5d6377eadb\") " Oct 30 21:41:54.819602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.819533 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:54.821907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.821851 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.822368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.822341 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "kube-api-access-76wlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:54.822650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.822621 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:54.823987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.823966 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:54.829111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.829084 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.834112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.834081 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config" pod "44562601-163d-400f-9394-8c5d6377eadb" (UID: "44562601-163d-400f-9394-8c5d6377eadb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:54.919481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919449 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.919481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919479 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.919481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919491 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wlr\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-kube-api-access-76wlr\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.919481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919501 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44562601-163d-400f-9394-8c5d6377eadb-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.919785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919531 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.919785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919539 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44562601-163d-400f-9394-8c5d6377eadb-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:54.919785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:54.919548 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44562601-163d-400f-9394-8c5d6377eadb-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:55.378088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.378045 4988 generic.go:356] "Generic (PLEG): container finished" podID="44562601-163d-400f-9394-8c5d6377eadb" containerID="680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b" exitCode=0 Oct 30 21:41:55.378293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.378088 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerDied","Data":"680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b"} Oct 30 21:41:55.378293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.378138 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:41:55.378293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.378159 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"44562601-163d-400f-9394-8c5d6377eadb","Type":"ContainerDied","Data":"6b70fc69f3ddea0f7ba61d03c2bebb161b89602633136d5f5252c6cf69d272d2"} Oct 30 21:41:55.378293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.378194 4988 scope.go:122] "RemoveContainer" containerID="59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab" Oct 30 21:41:55.388578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.388553 4988 scope.go:122] "RemoveContainer" containerID="680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b" Oct 30 21:41:55.396371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.396350 4988 scope.go:122] "RemoveContainer" containerID="7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458" Oct 30 21:41:55.404244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.404221 4988 scope.go:122] "RemoveContainer" containerID="59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab" Oct 30 21:41:55.404573 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:55.404552 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab\": container with ID starting with 59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab not found: ID does not exist" containerID="59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab" Oct 30 21:41:55.404641 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.404582 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab"} err="failed to get container status \"59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab\": rpc error: code = NotFound desc = could not find container \"59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab\": container with ID starting with 59279588b848ce2ac79ad4f02de1540c52631a7742fe4202e37d7d0ecb0bb8ab not found: ID does not exist" Oct 30 21:41:55.404641 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.404600 4988 scope.go:122] "RemoveContainer" containerID="680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b" Oct 30 21:41:55.404888 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:55.404870 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b\": container with ID starting with 680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b not found: ID does not exist" containerID="680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b" Oct 30 21:41:55.404944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.404892 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b"} err="failed to get container status \"680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b\": rpc error: code = NotFound desc = could not find container \"680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b\": container with ID starting with 680fe368908072d155416f2229b58c1181ff68fb2dfb66c1f251966f58357d2b not found: ID does not exist" Oct 30 21:41:55.404944 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.404903 4988 scope.go:122] "RemoveContainer" containerID="7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458" Oct 30 21:41:55.405163 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:55.405136 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458\": container with ID starting with 7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458 not found: ID does not exist" containerID="7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458" Oct 30 21:41:55.405201 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.405170 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458"} err="failed to get container status \"7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458\": rpc error: code = NotFound desc = could not find container \"7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458\": container with ID starting with 7cefa7faf730a616ac01eb8853e0db393f8c2333d4ec23271aca404161d9a458 not found: ID does not exist" Oct 30 21:41:55.543737 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.543702 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:41:55.548104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:55.548071 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:41:56.016663 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.016626 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" path="/var/lib/kubelet/pods/422b8946-a54d-47a5-8de7-66c7d6953723/volumes" Oct 30 21:41:56.017187 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.017171 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44562601-163d-400f-9394-8c5d6377eadb" path="/var/lib/kubelet/pods/44562601-163d-400f-9394-8c5d6377eadb/volumes" Oct 30 21:41:56.262946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.262904 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:56.263197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.263167 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="3b48d5cc-0671-4372-ad86-6eb593c320ad" containerName="otc-container" containerID="cri-o://84aa4d55b9b128c50711a9676a6832dd22f1fe7d2ff57667011d83c31553b791" gracePeriod=30 Oct 30 21:41:56.311330 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.311288 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:56.391035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.390857 4988 generic.go:356] "Generic (PLEG): container finished" podID="3b48d5cc-0671-4372-ad86-6eb593c320ad" containerID="84aa4d55b9b128c50711a9676a6832dd22f1fe7d2ff57667011d83c31553b791" exitCode=0 Oct 30 21:41:56.391035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.390973 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b48d5cc-0671-4372-ad86-6eb593c320ad","Type":"ContainerDied","Data":"84aa4d55b9b128c50711a9676a6832dd22f1fe7d2ff57667011d83c31553b791"} Oct 30 21:41:56.402113 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:41:56.402068 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:41:56.471698 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.471660 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:56.525557 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.525485 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:56.532602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.532565 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs\") pod \"3b48d5cc-0671-4372-ad86-6eb593c320ad\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " Oct 30 21:41:56.532841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.532615 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs\") pod \"3b48d5cc-0671-4372-ad86-6eb593c320ad\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " Oct 30 21:41:56.532841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.532641 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9\") pod \"3b48d5cc-0671-4372-ad86-6eb593c320ad\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " Oct 30 21:41:56.532841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.532699 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal\") pod \"3b48d5cc-0671-4372-ad86-6eb593c320ad\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " Oct 30 21:41:56.532841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.532728 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert\") pod \"3b48d5cc-0671-4372-ad86-6eb593c320ad\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " Oct 30 21:41:56.532841 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.532753 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs\") pod \"3b48d5cc-0671-4372-ad86-6eb593c320ad\" (UID: \"3b48d5cc-0671-4372-ad86-6eb593c320ad\") " Oct 30 21:41:56.533173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.533140 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal" pod "3b48d5cc-0671-4372-ad86-6eb593c320ad" (UID: "3b48d5cc-0671-4372-ad86-6eb593c320ad"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:56.535139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.535098 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs" pod "3b48d5cc-0671-4372-ad86-6eb593c320ad" (UID: "3b48d5cc-0671-4372-ad86-6eb593c320ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:56.535139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.535122 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs" pod "3b48d5cc-0671-4372-ad86-6eb593c320ad" (UID: "3b48d5cc-0671-4372-ad86-6eb593c320ad"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:56.535527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.535504 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert" pod "3b48d5cc-0671-4372-ad86-6eb593c320ad" (UID: "3b48d5cc-0671-4372-ad86-6eb593c320ad"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:56.535692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.535645 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9" pod "3b48d5cc-0671-4372-ad86-6eb593c320ad" (UID: "3b48d5cc-0671-4372-ad86-6eb593c320ad"). InnerVolumeSpecName "kube-api-access-75bq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:56.535692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.535645 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs" pod "3b48d5cc-0671-4372-ad86-6eb593c320ad" (UID: "3b48d5cc-0671-4372-ad86-6eb593c320ad"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:56.633328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.633285 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3b48d5cc-0671-4372-ad86-6eb593c320ad-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.633328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.633322 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.633328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.633332 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.633612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.633341 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.633612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.633351 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b48d5cc-0671-4372-ad86-6eb593c320ad-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.633612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.633360 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bq9\" (UniqueName: \"kubernetes.io/projected/3b48d5cc-0671-4372-ad86-6eb593c320ad-kube-api-access-75bq9\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:56.757005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:56.756965 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:57.396983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:57.396949 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:41:57.396983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:57.396968 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3b48d5cc-0671-4372-ad86-6eb593c320ad","Type":"ContainerDied","Data":"0f30f528b10237866851320bc6878cb2f1ef3e9281e8d634d3f75588e0121d72"} Oct 30 21:41:57.397576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:57.397027 4988 scope.go:122] "RemoveContainer" containerID="84aa4d55b9b128c50711a9676a6832dd22f1fe7d2ff57667011d83c31553b791" Oct 30 21:41:57.563189 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:57.563149 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:57.569226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:57.569189 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:41:57.991264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:57.991230 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.016776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:58.016735 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b48d5cc-0671-4372-ad86-6eb593c320ad" path="/var/lib/kubelet/pods/3b48d5cc-0671-4372-ad86-6eb593c320ad/volumes" Oct 30 21:41:58.175954 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:58.175919 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.351865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:58.351775 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.537604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:41:58.537549 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:03.637329 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:03.637288 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:11.705319 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:11.705280 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:11.903977 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:11.903932 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.065323 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.065224 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.225381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.225340 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.380236 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.380135 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.532961 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.532917 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.756001 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.755961 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww"] Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756255 4988 cpu_manager.go:414] "containerMap: removing container" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="method-gate" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756265 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="method-gate" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756277 4988 cpu_manager.go:414] "containerMap: removing container" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="prom-label-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756281 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="prom-label-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756291 4988 cpu_manager.go:414] "containerMap: removing container" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="prom-label-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756296 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="prom-label-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756306 4988 cpu_manager.go:414] "containerMap: removing container" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="init-config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756310 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="init-config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756316 4988 cpu_manager.go:414] "containerMap: removing container" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="thanos-sidecar" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756320 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="thanos-sidecar" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756329 4988 cpu_manager.go:414] "containerMap: removing container" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="prometheus" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756334 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="prometheus" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756347 4988 cpu_manager.go:414] "containerMap: removing container" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="kube-rbac-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756351 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="kube-rbac-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756357 4988 cpu_manager.go:414] "containerMap: removing container" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756361 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756368 4988 cpu_manager.go:414] "containerMap: removing container" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="init-config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756372 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="init-config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756382 4988 cpu_manager.go:414] "containerMap: removing container" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="alertmanager" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756387 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="alertmanager" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756394 4988 cpu_manager.go:414] "containerMap: removing container" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756398 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="config-reloader" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756405 4988 cpu_manager.go:414] "containerMap: removing container" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="method-gate" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756423 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="method-gate" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756430 4988 cpu_manager.go:414] "containerMap: removing container" podUID="3b48d5cc-0671-4372-ad86-6eb593c320ad" containerName="otc-container" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756439 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b48d5cc-0671-4372-ad86-6eb593c320ad" containerName="otc-container" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756450 4988 cpu_manager.go:414] "containerMap: removing container" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="kube-rbac-proxy" Oct 30 21:42:12.756449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756455 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="kube-rbac-proxy" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756504 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="method-gate" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756511 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b48d5cc-0671-4372-ad86-6eb593c320ad" containerName="otc-container" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756518 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="kube-rbac-proxy" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756524 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="method-gate" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756529 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="prom-label-proxy" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756534 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bb217e-6b85-4abf-8588-db4652cabe16" containerName="prom-label-proxy" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756540 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="thanos-sidecar" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756545 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="prometheus" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756551 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="08701e18-d0af-45b3-a571-274982048103" containerName="kube-rbac-proxy" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756556 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="config-reloader" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756562 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="44562601-163d-400f-9394-8c5d6377eadb" containerName="alertmanager" Oct 30 21:42:12.757437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.756568 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="422b8946-a54d-47a5-8de7-66c7d6953723" containerName="config-reloader" Oct 30 21:42:12.761182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.761161 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.835673 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.835632 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:42:12.852691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.852656 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:12.862746 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.862713 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.862872 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.862754 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfjh\" (UniqueName: \"kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.862872 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.862771 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.863036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.862911 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.863036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.862946 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.863170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.863114 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:42:12.863170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.863122 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:42:12.863170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.863166 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:42:12.863331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.863173 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:42:12.863331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.863201 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-5hmf6\"" Oct 30 21:42:12.863331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.863212 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:42:12.864512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.864492 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww"] Oct 30 21:42:12.870979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.870953 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5"] Oct 30 21:42:12.875990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.875971 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:12.928477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.928446 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:42:12.928673 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.928479 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:42:12.928673 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.928542 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-pt7mz\"" Oct 30 21:42:12.934185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.934148 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5"] Oct 30 21:42:12.963615 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963566 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfjh\" (UniqueName: \"kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.963615 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963616 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.963901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963735 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:12.963901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963792 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.963901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963857 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.963901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963890 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqq7q\" (UniqueName: \"kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:12.964124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963947 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.964124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.963972 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:12.964495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.964472 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.964622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.964558 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.964688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.964652 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:12.966285 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:12.966263 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:13.025077 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.024990 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfjh\" (UniqueName: \"kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:13.057077 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.057035 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:13.065048 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.065006 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqq7q\" (UniqueName: \"kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.065185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.065074 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.065185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.065105 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.065843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.065819 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.067751 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.067724 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.079477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.079442 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:42:13.126966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.126932 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqq7q\" (UniqueName: \"kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q\") pod \"data-science-thanos-querier-proxy-77657866bd-m7bt5\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.193226 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.193187 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:42:13.465428 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.465383 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww"] Oct 30 21:42:13.468843 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:42:13.468811 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb468b352_dabe_461d_b88b_bbd0acec2ef8.slice/crio-3316fd9c9197f32b0198e0c5a565a732c3402a4740f6163cf797b4a4efafb544 WatchSource:0}: Error finding container 3316fd9c9197f32b0198e0c5a565a732c3402a4740f6163cf797b4a4efafb544: Status 404 returned error can't find the container with id 3316fd9c9197f32b0198e0c5a565a732c3402a4740f6163cf797b4a4efafb544 Oct 30 21:42:13.580482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:13.580449 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5"] Oct 30 21:42:13.584437 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:42:13.584380 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad25116_95f1_4022_8053_359a875baae4.slice/crio-5230c7095a3956e05effd8e8dfd5c31ee6f1522bcb05fcf3ffaedb167dddc21f WatchSource:0}: Error finding container 5230c7095a3956e05effd8e8dfd5c31ee6f1522bcb05fcf3ffaedb167dddc21f: Status 404 returned error can't find the container with id 5230c7095a3956e05effd8e8dfd5c31ee6f1522bcb05fcf3ffaedb167dddc21f Oct 30 21:42:14.365589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.365552 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:14.478614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.478570 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerStarted","Data":"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05"} Oct 30 21:42:14.478813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.478647 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerStarted","Data":"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871"} Oct 30 21:42:14.478813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.478681 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerStarted","Data":"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772"} Oct 30 21:42:14.478813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.478706 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerStarted","Data":"3316fd9c9197f32b0198e0c5a565a732c3402a4740f6163cf797b4a4efafb544"} Oct 30 21:42:14.480458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.480402 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerStarted","Data":"bb661a4a6f46b02efaa069347f0487be11c0627d307952bc7ca0755f51e5e21e"} Oct 30 21:42:14.480589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.480484 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerStarted","Data":"c06e3b6765e03e2a5df3ddec14e9854f714698df02c121b40c5fcc31ce88158d"} Oct 30 21:42:14.480589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.480512 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerStarted","Data":"2ce32181f3382ea92de6a03ecbc9d5e392134e77c3a9ebba418fed43f3645499"} Oct 30 21:42:14.480589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.480533 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerStarted","Data":"5230c7095a3956e05effd8e8dfd5c31ee6f1522bcb05fcf3ffaedb167dddc21f"} Oct 30 21:42:14.605280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.605215 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" podStartSLOduration=2.605201476 podStartE2EDuration="2.605201476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:14.604895684 +0000 UTC m=+3962.712391596" watchObservedRunningTime="2026-10-30 21:42:14.605201476 +0000 UTC m=+3962.712697382" Oct 30 21:42:14.723890 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:14.723834 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" podStartSLOduration=2.723819871 podStartE2EDuration="2.723819871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:14.722207324 +0000 UTC m=+3962.829703246" watchObservedRunningTime="2026-10-30 21:42:14.723819871 +0000 UTC m=+3962.831315781" Oct 30 21:42:17.526063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.526028 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:17.568249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.568209 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:42:17.572148 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.572115 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.608310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.608269 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.608516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.608326 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.608516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.608428 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfmm\" (UniqueName: \"kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.608516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.608484 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.608516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.608505 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.608701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.608537 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.640136 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.640093 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:42:17.659739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.659703 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:42:17.677131 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.677091 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:42:17.677317 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.677157 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:42:17.677317 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.677239 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:42:17.687667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.687639 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-s9tf6\"" Oct 30 21:42:17.688398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.688379 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:42:17.704058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.704023 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:17.709658 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.709632 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.709776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.709672 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.709776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.709696 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfmm\" (UniqueName: \"kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.709776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.709722 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.709776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.709747 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.710001 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.709791 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.710515 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:42:17.710115 4988 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 21:42:17.710515 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:42:17.710185 4988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs podName:c575af4a-9323-409b-8849-e48b706b7ac6 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:18.210164944 +0000 UTC m=+3966.317660847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6") : secret "data-science-collector-collector-headless-tls" not found Oct 30 21:42:17.710722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.710532 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.712528 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.712510 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.712620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.712558 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.712620 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.712600 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.781512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.781397 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfmm\" (UniqueName: \"kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:17.889306 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:17.889268 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:18.045249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.045159 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:18.213404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.213354 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:18.216056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.216024 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:18.490519 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.490397 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:42:18.867535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.867447 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:42:18.870397 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:42:18.870367 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc575af4a_9323_409b_8849_e48b706b7ac6.slice/crio-baba9a2066d147c478fcad3cc3816e00f079990dcd352f65388de039f7827847 WatchSource:0}: Error finding container baba9a2066d147c478fcad3cc3816e00f079990dcd352f65388de039f7827847: Status 404 returned error can't find the container with id baba9a2066d147c478fcad3cc3816e00f079990dcd352f65388de039f7827847 Oct 30 21:42:18.872493 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.872459 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:18.872583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:18.872536 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:19.503988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:19.503952 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c575af4a-9323-409b-8849-e48b706b7ac6","Type":"ContainerStarted","Data":"fc838f7395ffef69a1d53db743215be48d120632e77a7df3f6ed8ce612a72f54"} Oct 30 21:42:19.504150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:19.504029 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c575af4a-9323-409b-8849-e48b706b7ac6","Type":"ContainerStarted","Data":"baba9a2066d147c478fcad3cc3816e00f079990dcd352f65388de039f7827847"} Oct 30 21:42:19.672528 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:19.672463 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.672448236 podStartE2EDuration="2.672448236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:19.621797612 +0000 UTC m=+3967.729293522" watchObservedRunningTime="2026-10-30 21:42:19.672448236 +0000 UTC m=+3967.779944147" Oct 30 21:42:24.122262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:24.122224 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:26.394095 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:42:26.394054 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:42:29.981667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:29.981627 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.326372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:30.326290 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.483062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:30.483019 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:30.709554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:30.709515 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.243924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.243891 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.457491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.457450 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:31.458987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.458965 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:42:31.463097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.463074 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.492250 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.492208 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:42:31.540725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.540633 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:42:31.540958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.540638 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:42:31.540958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.540677 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:42:31.541108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.540685 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-rfvbp\"" Oct 30 21:42:31.541108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.540688 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:42:31.541108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.540746 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:42:31.541888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.541869 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:42:31.545875 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.545853 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.573302 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.573264 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:42:31.596202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.596164 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:42:31.611257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.611224 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:42:31.611800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.611773 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:42:31.611938 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.611863 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:42:31.625844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.625811 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.625861 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.625942 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.625998 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626024 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626106 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626142 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626163 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626179 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626200 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxgw\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626253 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.626346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.626286 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.632008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.631973 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:42:31.632008 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.631973 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:42:31.632262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.632103 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:42:31.632262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.632115 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:42:31.632262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.632119 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-crhsh\"" Oct 30 21:42:31.726665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.726631 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.726868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.726847 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.726954 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.726897 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.727005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.726973 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.727114 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727094 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.727183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727162 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.727242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727206 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.727302 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727237 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.727302 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727266 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864lx\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.727652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727335 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.727761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727736 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.727953 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727927 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.727979 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728009 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxgw\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728156 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728042 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728156 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728072 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728156 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728105 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.728156 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728152 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728201 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.728379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.728228 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.730174 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.730144 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.730862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.730302 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.731020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.730747 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.731145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.730757 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.731291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.731265 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.731382 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.731358 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.731910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.731886 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.732722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.732695 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.732818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.732801 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.733216 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.733199 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.790284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.790244 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxgw\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:31.829204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829108 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829157 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829184 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829210 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864lx\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829261 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829340 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829361 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.829779 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.829682 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.831570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.831538 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.831825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.831801 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.831914 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.831806 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.832015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.831999 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.832052 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.832017 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:31.891457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:31.891423 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864lx\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:32.167896 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:32.167851 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:32.599581 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:32.599383 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:42:32.602311 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:42:32.602282 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b020adf_0e54_4e96_a67d_56a593f1137d.slice/crio-87a0ec1965b7719c881cf2d67037091238d2a35d5868ee8db3e63b965c7357c4 WatchSource:0}: Error finding container 87a0ec1965b7719c881cf2d67037091238d2a35d5868ee8db3e63b965c7357c4: Status 404 returned error can't find the container with id 87a0ec1965b7719c881cf2d67037091238d2a35d5868ee8db3e63b965c7357c4 Oct 30 21:42:33.568361 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:33.568322 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerStarted","Data":"33f973591d3cb9cdc86039e1bfe8f4573f506419bf1d4e5eef36ff903b086346"} Oct 30 21:42:33.568602 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:33.568397 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerStarted","Data":"87a0ec1965b7719c881cf2d67037091238d2a35d5868ee8db3e63b965c7357c4"} Oct 30 21:42:36.065737 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.065695 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.170358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.170324 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.267580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.267538 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.267782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.267696 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.320261 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.320177 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.462482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.462446 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.529552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.529522 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:36.582722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.582632 4988 generic.go:356] "Generic (PLEG): container finished" podID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerID="33f973591d3cb9cdc86039e1bfe8f4573f506419bf1d4e5eef36ff903b086346" exitCode=0 Oct 30 21:42:36.582722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.582705 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerDied","Data":"33f973591d3cb9cdc86039e1bfe8f4573f506419bf1d4e5eef36ff903b086346"} Oct 30 21:42:36.583399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:36.583379 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:37.026315 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.026210 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:42:37.028947 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:42:37.028918 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5888c9d7_012d_4e75_a707_71f67d648ee8.slice/crio-aff6fdfd3381fa0b9bd2d17bcc6a8b15259f958357274a06240be620e2c1b34d WatchSource:0}: Error finding container aff6fdfd3381fa0b9bd2d17bcc6a8b15259f958357274a06240be620e2c1b34d: Status 404 returned error can't find the container with id aff6fdfd3381fa0b9bd2d17bcc6a8b15259f958357274a06240be620e2c1b34d Oct 30 21:42:37.589421 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.589377 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerStarted","Data":"500074f341a5cbcef6701718a484023ac74f31e228de7840a7dbbbe48a32f015"} Oct 30 21:42:37.589918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.589464 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerStarted","Data":"0ed50966c793aab8210303e4ff6c965375f353fd50a909f333a37a49e5b86e4b"} Oct 30 21:42:37.589918 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.589660 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:37.591031 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.590998 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerStarted","Data":"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b"} Oct 30 21:42:37.591149 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.591045 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerStarted","Data":"aff6fdfd3381fa0b9bd2d17bcc6a8b15259f958357274a06240be620e2c1b34d"} Oct 30 21:42:37.713745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:37.713687 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.33053046 podStartE2EDuration="6.713673042s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.383142582s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:37.711938148 +0000 UTC m=+3985.819434084" watchObservedRunningTime="2026-10-30 21:42:37.713673042 +0000 UTC m=+3985.821168952" Oct 30 21:42:41.610730 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:41.610692 4988 generic.go:356] "Generic (PLEG): container finished" podID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerID="ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b" exitCode=0 Oct 30 21:42:41.611145 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:41.610771 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerDied","Data":"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b"} Oct 30 21:42:42.618902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:42.618819 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerStarted","Data":"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1"} Oct 30 21:42:42.618902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:42.618887 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerStarted","Data":"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb"} Oct 30 21:42:42.619318 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:42.618912 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerStarted","Data":"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7"} Oct 30 21:42:42.747712 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:42.747653 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=8.161436805 podStartE2EDuration="11.747637962s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586201157s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:42.74621661 +0000 UTC m=+3990.853712533" watchObservedRunningTime="2026-10-30 21:42:42.747637962 +0000 UTC m=+3990.855133874" Oct 30 21:42:43.597650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:43.597611 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:42:46.583952 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:46.583914 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:48.588087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:48.588047 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.807018 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:48.806979 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:51.584484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:51.584443 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:51.589690 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:51.589666 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:51.664025 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:42:51.663993 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:42:56.394236 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:42:56.394162 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:43:09.656470 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:09.656433 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.807260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:09.807222 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.961517 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:09.961433 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:10.218882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:10.218783 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:10.380720 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:10.380685 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:10.531526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:10.531429 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.443710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:15.443675 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:15.444157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:15.443859 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c575af4a-9323-409b-8849-e48b706b7ac6" containerName="otc-container" containerID="cri-o://fc838f7395ffef69a1d53db743215be48d120632e77a7df3f6ed8ce612a72f54" gracePeriod=30 Oct 30 21:43:15.781401 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:15.781304 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:15.942862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:15.942824 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:16.135459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:16.135399 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.786320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.786289 4988 generic.go:356] "Generic (PLEG): container finished" podID="c575af4a-9323-409b-8849-e48b706b7ac6" containerID="fc838f7395ffef69a1d53db743215be48d120632e77a7df3f6ed8ce612a72f54" exitCode=0 Oct 30 21:43:17.786733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.786371 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c575af4a-9323-409b-8849-e48b706b7ac6","Type":"ContainerDied","Data":"fc838f7395ffef69a1d53db743215be48d120632e77a7df3f6ed8ce612a72f54"} Oct 30 21:43:17.786733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.786444 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c575af4a-9323-409b-8849-e48b706b7ac6","Type":"ContainerDied","Data":"baba9a2066d147c478fcad3cc3816e00f079990dcd352f65388de039f7827847"} Oct 30 21:43:17.786733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.786463 4988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baba9a2066d147c478fcad3cc3816e00f079990dcd352f65388de039f7827847" Oct 30 21:43:17.790044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.790023 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:17.826897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.826857 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs\") pod \"c575af4a-9323-409b-8849-e48b706b7ac6\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " Oct 30 21:43:17.827097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.826907 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal\") pod \"c575af4a-9323-409b-8849-e48b706b7ac6\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " Oct 30 21:43:17.827097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.826939 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs\") pod \"c575af4a-9323-409b-8849-e48b706b7ac6\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " Oct 30 21:43:17.827097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.826976 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\") pod \"c575af4a-9323-409b-8849-e48b706b7ac6\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " Oct 30 21:43:17.827097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.827010 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm\" (UniqueName: \"kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm\") pod \"c575af4a-9323-409b-8849-e48b706b7ac6\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " Oct 30 21:43:17.827097 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.827077 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert\") pod \"c575af4a-9323-409b-8849-e48b706b7ac6\" (UID: \"c575af4a-9323-409b-8849-e48b706b7ac6\") " Oct 30 21:43:17.827433 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.827380 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal" pod "c575af4a-9323-409b-8849-e48b706b7ac6" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:17.829300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.829265 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs" pod "c575af4a-9323-409b-8849-e48b706b7ac6" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:17.829475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.829269 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm" pod "c575af4a-9323-409b-8849-e48b706b7ac6" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6"). InnerVolumeSpecName "kube-api-access-2rfmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:17.829475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.829334 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert" pod "c575af4a-9323-409b-8849-e48b706b7ac6" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:17.829610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.829539 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs" pod "c575af4a-9323-409b-8849-e48b706b7ac6" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:17.829610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.829555 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs" pod "c575af4a-9323-409b-8849-e48b706b7ac6" (UID: "c575af4a-9323-409b-8849-e48b706b7ac6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:17.927798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.927762 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:17.927798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.927793 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:17.927798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.927804 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c575af4a-9323-409b-8849-e48b706b7ac6-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:17.928050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.927814 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:17.928050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.927823 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c575af4a-9323-409b-8849-e48b706b7ac6-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:17.928050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:17.927831 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rfmm\" (UniqueName: \"kubernetes.io/projected/c575af4a-9323-409b-8849-e48b706b7ac6-kube-api-access-2rfmm\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:18.791774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:18.791738 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:18.986638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:18.986601 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:19.026960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.026927 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:19.077869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.077787 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:19.078183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.078169 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c575af4a-9323-409b-8849-e48b706b7ac6" containerName="otc-container" Oct 30 21:43:19.078230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.078185 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c575af4a-9323-409b-8849-e48b706b7ac6" containerName="otc-container" Oct 30 21:43:19.078264 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.078255 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c575af4a-9323-409b-8849-e48b706b7ac6" containerName="otc-container" Oct 30 21:43:19.081301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.081280 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.134820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.134779 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:43:19.134820 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.134779 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:43:19.136469 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136446 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:43:19.136598 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136571 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.136671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136645 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.136671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136646 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:43:19.136783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136725 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.136783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136740 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:43:19.136897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136782 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.136897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136838 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrm6\" (UniqueName: \"kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.136897 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.136879 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.137111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.137096 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-s9tf6\"" Oct 30 21:43:19.185997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.185951 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:19.237228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.237188 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.237228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.237232 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.237532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.237271 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.237532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.237301 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrm6\" (UniqueName: \"kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.237532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.237328 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.237532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.237374 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.238127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.238096 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.239959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.239930 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.240073 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.239935 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.240206 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.240188 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.240274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.240207 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.299678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.299641 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrm6\" (UniqueName: \"kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6\") pod \"data-science-collector-collector-1\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.393900 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.393867 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:19.770859 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.770643 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:19.773543 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:43:19.773516 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563d540c_97e6_451e_bf59_c1e9736f0bd9.slice/crio-0713166e3277aa4ff5e9b0ff1ceeed58436ab13cf76f1492845df3ec8016b069 WatchSource:0}: Error finding container 0713166e3277aa4ff5e9b0ff1ceeed58436ab13cf76f1492845df3ec8016b069: Status 404 returned error can't find the container with id 0713166e3277aa4ff5e9b0ff1ceeed58436ab13cf76f1492845df3ec8016b069 Oct 30 21:43:19.775597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.775560 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:43:19.775714 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.775644 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:43:19.797312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:19.797271 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"563d540c-97e6-451e-bf59-c1e9736f0bd9","Type":"ContainerStarted","Data":"0713166e3277aa4ff5e9b0ff1ceeed58436ab13cf76f1492845df3ec8016b069"} Oct 30 21:43:20.015825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:20.015783 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c575af4a-9323-409b-8849-e48b706b7ac6" path="/var/lib/kubelet/pods/c575af4a-9323-409b-8849-e48b706b7ac6/volumes" Oct 30 21:43:20.803749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:20.803711 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"563d540c-97e6-451e-bf59-c1e9736f0bd9","Type":"ContainerStarted","Data":"272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295"} Oct 30 21:43:20.933012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:20.932955 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.932937834 podStartE2EDuration="1.932937834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:20.93168393 +0000 UTC m=+4029.039179844" watchObservedRunningTime="2026-10-30 21:43:20.932937834 +0000 UTC m=+4029.040433744" Oct 30 21:43:23.527169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:23.527129 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:24.428632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:24.428591 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:26.393488 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:43:26.393448 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:43:41.443679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:41.443637 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:41.604612 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:41.604566 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:41.775466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:41.775358 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:41.982848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:41.982797 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:42.166049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:42.166016 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:42.390902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:42.390868 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.214596 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:47.214553 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:47.215040 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:47.214786 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="563d540c-97e6-451e-bf59-c1e9736f0bd9" containerName="otc-container" containerID="cri-o://272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295" gracePeriod=30 Oct 30 21:43:47.582460 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:47.582367 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.772116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:47.772074 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.948782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:47.948741 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:49.561281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.561251 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:49.709290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709252 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal\") pod \"563d540c-97e6-451e-bf59-c1e9736f0bd9\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " Oct 30 21:43:49.709512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709334 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6\" (UniqueName: \"kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6\") pod \"563d540c-97e6-451e-bf59-c1e9736f0bd9\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " Oct 30 21:43:49.709512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709433 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs\") pod \"563d540c-97e6-451e-bf59-c1e9736f0bd9\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " Oct 30 21:43:49.709512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709469 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert\") pod \"563d540c-97e6-451e-bf59-c1e9736f0bd9\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " Oct 30 21:43:49.709683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709528 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs\") pod \"563d540c-97e6-451e-bf59-c1e9736f0bd9\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " Oct 30 21:43:49.709683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709564 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs\") pod \"563d540c-97e6-451e-bf59-c1e9736f0bd9\" (UID: \"563d540c-97e6-451e-bf59-c1e9736f0bd9\") " Oct 30 21:43:49.709813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.709777 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal" pod "563d540c-97e6-451e-bf59-c1e9736f0bd9" (UID: "563d540c-97e6-451e-bf59-c1e9736f0bd9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:49.711919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.711878 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs" pod "563d540c-97e6-451e-bf59-c1e9736f0bd9" (UID: "563d540c-97e6-451e-bf59-c1e9736f0bd9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:49.712155 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.712127 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6" pod "563d540c-97e6-451e-bf59-c1e9736f0bd9" (UID: "563d540c-97e6-451e-bf59-c1e9736f0bd9"). InnerVolumeSpecName "kube-api-access-psrm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:49.712257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.712191 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs" pod "563d540c-97e6-451e-bf59-c1e9736f0bd9" (UID: "563d540c-97e6-451e-bf59-c1e9736f0bd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:49.712351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.712316 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs" pod "563d540c-97e6-451e-bf59-c1e9736f0bd9" (UID: "563d540c-97e6-451e-bf59-c1e9736f0bd9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:49.712439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.712382 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert" pod "563d540c-97e6-451e-bf59-c1e9736f0bd9" (UID: "563d540c-97e6-451e-bf59-c1e9736f0bd9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:49.811072 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.811031 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:49.811072 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.811067 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:49.811072 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.811077 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:49.811346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.811087 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/563d540c-97e6-451e-bf59-c1e9736f0bd9-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:49.811346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.811096 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/563d540c-97e6-451e-bf59-c1e9736f0bd9-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:49.811346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.811105 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psrm6\" (UniqueName: \"kubernetes.io/projected/563d540c-97e6-451e-bf59-c1e9736f0bd9-kube-api-access-psrm6\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:49.938880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.938844 4988 generic.go:356] "Generic (PLEG): container finished" podID="563d540c-97e6-451e-bf59-c1e9736f0bd9" containerID="272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295" exitCode=0 Oct 30 21:43:49.939042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.938902 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:49.939042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.938924 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"563d540c-97e6-451e-bf59-c1e9736f0bd9","Type":"ContainerDied","Data":"272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295"} Oct 30 21:43:49.939042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.939011 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"563d540c-97e6-451e-bf59-c1e9736f0bd9","Type":"ContainerDied","Data":"0713166e3277aa4ff5e9b0ff1ceeed58436ab13cf76f1492845df3ec8016b069"} Oct 30 21:43:49.939042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.939039 4988 scope.go:122] "RemoveContainer" containerID="272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295" Oct 30 21:43:49.948578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.948509 4988 scope.go:122] "RemoveContainer" containerID="272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295" Oct 30 21:43:49.948866 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:43:49.948844 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295\": container with ID starting with 272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295 not found: ID does not exist" containerID="272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295" Oct 30 21:43:49.948912 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:49.948876 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295"} err="failed to get container status \"272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295\": rpc error: code = NotFound desc = could not find container \"272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295\": container with ID starting with 272b996783e45cfa66dc32767a56fa94dff422d34ffb1a7baae8c59234a25295 not found: ID does not exist" Oct 30 21:43:50.131380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.131285 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:50.181029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.180985 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:50.219805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.219770 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:50.220230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.220212 4988 cpu_manager.go:414] "containerMap: removing container" podUID="563d540c-97e6-451e-bf59-c1e9736f0bd9" containerName="otc-container" Oct 30 21:43:50.220327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.220232 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563d540c-97e6-451e-bf59-c1e9736f0bd9" containerName="otc-container" Oct 30 21:43:50.220385 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.220354 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="563d540c-97e6-451e-bf59-c1e9736f0bd9" containerName="otc-container" Oct 30 21:43:50.224801 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.224778 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.281706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.281676 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:43:50.281891 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.281720 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:43:50.281945 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.281915 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:43:50.282004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.281985 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:43:50.282051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.282039 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-s9tf6\"" Oct 30 21:43:50.282094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.282059 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:43:50.286641 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.286618 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:50.316625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.316582 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.316813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.316633 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9b4\" (UniqueName: \"kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.316813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.316660 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.316813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.316782 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.316942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.316810 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.316942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.316835 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418051 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418098 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9b4\" (UniqueName: \"kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418224 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418281 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418305 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418342 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.418894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.418860 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.420885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.420859 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.420885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.420875 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.421126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.421107 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.421185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.421108 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.492229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.492189 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9b4\" (UniqueName: \"kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4\") pod \"data-science-collector-collector-1\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.537777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.537745 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:43:50.925591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.925568 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:43:50.927878 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:43:50.927846 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd73a97_1a06_45c7_b250_3ca9c7579e36.slice/crio-19266e3448cdfc1c6bc9813089498a3a324440646726e976d59a7306a1ae37b2 WatchSource:0}: Error finding container 19266e3448cdfc1c6bc9813089498a3a324440646726e976d59a7306a1ae37b2: Status 404 returned error can't find the container with id 19266e3448cdfc1c6bc9813089498a3a324440646726e976d59a7306a1ae37b2 Oct 30 21:43:50.929979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.929938 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:43:50.930092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.930018 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:43:50.946112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:50.946080 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0fd73a97-1a06-45c7-b250-3ca9c7579e36","Type":"ContainerStarted","Data":"19266e3448cdfc1c6bc9813089498a3a324440646726e976d59a7306a1ae37b2"} Oct 30 21:43:51.951904 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:51.951869 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0fd73a97-1a06-45c7-b250-3ca9c7579e36","Type":"ContainerStarted","Data":"4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab"} Oct 30 21:43:52.016209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:52.016170 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563d540c-97e6-451e-bf59-c1e9736f0bd9" path="/var/lib/kubelet/pods/563d540c-97e6-451e-bf59-c1e9736f0bd9/volumes" Oct 30 21:43:52.086385 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:43:52.086325 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.086308576 podStartE2EDuration="2.086308576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:52.08379734 +0000 UTC m=+4060.191293250" watchObservedRunningTime="2026-10-30 21:43:52.086308576 +0000 UTC m=+4060.193804489" Oct 30 21:43:56.394337 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:43:56.394289 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:44:04.714212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.714174 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww"] Oct 30 21:44:04.714713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.714466 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="prom-label-proxy" containerID="cri-o://82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" gracePeriod=30 Oct 30 21:44:04.714713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.714474 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="method-gate" containerID="cri-o://06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" gracePeriod=30 Oct 30 21:44:04.714713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.714470 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="kube-rbac-proxy" containerID="cri-o://23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" gracePeriod=30 Oct 30 21:44:04.818191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.818150 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5"] Oct 30 21:44:04.818452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.818424 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="kube-rbac-proxy" containerID="cri-o://2ce32181f3382ea92de6a03ecbc9d5e392134e77c3a9ebba418fed43f3645499" gracePeriod=30 Oct 30 21:44:04.818524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.818441 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="prom-label-proxy" containerID="cri-o://bb661a4a6f46b02efaa069347f0487be11c0627d307952bc7ca0755f51e5e21e" gracePeriod=30 Oct 30 21:44:04.818524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.818464 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="method-gate" containerID="cri-o://c06e3b6765e03e2a5df3ddec14e9854f714698df02c121b40c5fcc31ce88158d" gracePeriod=30 Oct 30 21:44:04.976617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:04.976583 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:44:05.016177 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.016136 4988 generic.go:356] "Generic (PLEG): container finished" podID="2ad25116-95f1-4022-8053-359a875baae4" containerID="bb661a4a6f46b02efaa069347f0487be11c0627d307952bc7ca0755f51e5e21e" exitCode=0 Oct 30 21:44:05.016177 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.016174 4988 generic.go:356] "Generic (PLEG): container finished" podID="2ad25116-95f1-4022-8053-359a875baae4" containerID="c06e3b6765e03e2a5df3ddec14e9854f714698df02c121b40c5fcc31ce88158d" exitCode=0 Oct 30 21:44:05.016396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.016183 4988 generic.go:356] "Generic (PLEG): container finished" podID="2ad25116-95f1-4022-8053-359a875baae4" containerID="2ce32181f3382ea92de6a03ecbc9d5e392134e77c3a9ebba418fed43f3645499" exitCode=0 Oct 30 21:44:05.016396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.016208 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerDied","Data":"bb661a4a6f46b02efaa069347f0487be11c0627d307952bc7ca0755f51e5e21e"} Oct 30 21:44:05.016396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.016270 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerDied","Data":"c06e3b6765e03e2a5df3ddec14e9854f714698df02c121b40c5fcc31ce88158d"} Oct 30 21:44:05.016396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.016299 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerDied","Data":"2ce32181f3382ea92de6a03ecbc9d5e392134e77c3a9ebba418fed43f3645499"} Oct 30 21:44:05.018224 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018195 4988 generic.go:356] "Generic (PLEG): container finished" podID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerID="82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" exitCode=0 Oct 30 21:44:05.018224 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018222 4988 generic.go:356] "Generic (PLEG): container finished" podID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerID="06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" exitCode=0 Oct 30 21:44:05.018442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018232 4988 generic.go:356] "Generic (PLEG): container finished" podID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerID="23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" exitCode=0 Oct 30 21:44:05.018442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018232 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerDied","Data":"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05"} Oct 30 21:44:05.018442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018338 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerDied","Data":"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871"} Oct 30 21:44:05.018442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018353 4988 scope.go:122] "RemoveContainer" containerID="82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" Oct 30 21:44:05.018442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018382 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerDied","Data":"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772"} Oct 30 21:44:05.018442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018436 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" event={"ID":"b468b352-dabe-461d-b88b-bbd0acec2ef8","Type":"ContainerDied","Data":"3316fd9c9197f32b0198e0c5a565a732c3402a4740f6163cf797b4a4efafb544"} Oct 30 21:44:05.018664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.018263 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww" Oct 30 21:44:05.028491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.028464 4988 scope.go:122] "RemoveContainer" containerID="06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" Oct 30 21:44:05.039016 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.038987 4988 scope.go:122] "RemoveContainer" containerID="23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" Oct 30 21:44:05.048593 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.048569 4988 scope.go:122] "RemoveContainer" containerID="82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" Oct 30 21:44:05.048942 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:05.048922 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": container with ID starting with 82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05 not found: ID does not exist" containerID="82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" Oct 30 21:44:05.048999 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.048953 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05"} err="failed to get container status \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": rpc error: code = NotFound desc = could not find container \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": container with ID starting with 82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05 not found: ID does not exist" Oct 30 21:44:05.048999 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.048971 4988 scope.go:122] "RemoveContainer" containerID="06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" Oct 30 21:44:05.049204 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:05.049189 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": container with ID starting with 06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871 not found: ID does not exist" containerID="06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" Oct 30 21:44:05.049247 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049206 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871"} err="failed to get container status \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": rpc error: code = NotFound desc = could not find container \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": container with ID starting with 06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871 not found: ID does not exist" Oct 30 21:44:05.049247 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049218 4988 scope.go:122] "RemoveContainer" containerID="23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" Oct 30 21:44:05.049440 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:05.049404 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": container with ID starting with 23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772 not found: ID does not exist" containerID="23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" Oct 30 21:44:05.049491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049445 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772"} err="failed to get container status \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": rpc error: code = NotFound desc = could not find container \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": container with ID starting with 23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772 not found: ID does not exist" Oct 30 21:44:05.049491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049458 4988 scope.go:122] "RemoveContainer" containerID="82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" Oct 30 21:44:05.049739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049719 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05"} err="failed to get container status \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": rpc error: code = NotFound desc = could not find container \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": container with ID starting with 82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05 not found: ID does not exist" Oct 30 21:44:05.049795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049740 4988 scope.go:122] "RemoveContainer" containerID="06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" Oct 30 21:44:05.049958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049943 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871"} err="failed to get container status \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": rpc error: code = NotFound desc = could not find container \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": container with ID starting with 06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871 not found: ID does not exist" Oct 30 21:44:05.049958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.049957 4988 scope.go:122] "RemoveContainer" containerID="23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" Oct 30 21:44:05.050163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050138 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772"} err="failed to get container status \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": rpc error: code = NotFound desc = could not find container \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": container with ID starting with 23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772 not found: ID does not exist" Oct 30 21:44:05.050241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050166 4988 scope.go:122] "RemoveContainer" containerID="82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05" Oct 30 21:44:05.050499 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050478 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05"} err="failed to get container status \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": rpc error: code = NotFound desc = could not find container \"82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05\": container with ID starting with 82075bc11a1cd16813b4f21c65bc7b8cdb38290ff44101e5839ae585f5b94d05 not found: ID does not exist" Oct 30 21:44:05.050553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050500 4988 scope.go:122] "RemoveContainer" containerID="06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871" Oct 30 21:44:05.050740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050719 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871"} err="failed to get container status \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": rpc error: code = NotFound desc = could not find container \"06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871\": container with ID starting with 06a44029fd7e09a9a5ed0906f9a0a526cea6361f216160a179d6932007e0d871 not found: ID does not exist" Oct 30 21:44:05.050827 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050742 4988 scope.go:122] "RemoveContainer" containerID="23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772" Oct 30 21:44:05.051017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.050996 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772"} err="failed to get container status \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": rpc error: code = NotFound desc = could not find container \"23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772\": container with ID starting with 23ddc108e7fd75f1a09619d0f5694242c7daf8ff99f896657261502099a49772 not found: ID does not exist" Oct 30 21:44:05.052279 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052260 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh\" (UniqueName: \"kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh\") pod \"b468b352-dabe-461d-b88b-bbd0acec2ef8\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " Oct 30 21:44:05.052349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052301 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls\" (UniqueName: \"kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls\") pod \"b468b352-dabe-461d-b88b-bbd0acec2ef8\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " Oct 30 21:44:05.052349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052326 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config\") pod \"b468b352-dabe-461d-b88b-bbd0acec2ef8\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " Oct 30 21:44:05.052472 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052454 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca\") pod \"b468b352-dabe-461d-b88b-bbd0acec2ef8\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " Oct 30 21:44:05.052539 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052514 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca\") pod \"b468b352-dabe-461d-b88b-bbd0acec2ef8\" (UID: \"b468b352-dabe-461d-b88b-bbd0acec2ef8\") " Oct 30 21:44:05.052782 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052757 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config" pod "b468b352-dabe-461d-b88b-bbd0acec2ef8" (UID: "b468b352-dabe-461d-b88b-bbd0acec2ef8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:05.052903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052855 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca" pod "b468b352-dabe-461d-b88b-bbd0acec2ef8" (UID: "b468b352-dabe-461d-b88b-bbd0acec2ef8"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:05.052968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.052899 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca" pod "b468b352-dabe-461d-b88b-bbd0acec2ef8" (UID: "b468b352-dabe-461d-b88b-bbd0acec2ef8"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:05.054772 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.054746 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh" pod "b468b352-dabe-461d-b88b-bbd0acec2ef8" (UID: "b468b352-dabe-461d-b88b-bbd0acec2ef8"). InnerVolumeSpecName "kube-api-access-vjfjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:05.054883 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.054780 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls" pod "b468b352-dabe-461d-b88b-bbd0acec2ef8" (UID: "b468b352-dabe-461d-b88b-bbd0acec2ef8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:05.079998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.079961 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:44:05.153197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153163 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q\" (UniqueName: \"kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q\") pod \"2ad25116-95f1-4022-8053-359a875baae4\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " Oct 30 21:44:05.153371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153230 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config\" (UniqueName: \"kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config\") pod \"2ad25116-95f1-4022-8053-359a875baae4\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " Oct 30 21:44:05.153371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153262 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls\" (UniqueName: \"kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls\") pod \"2ad25116-95f1-4022-8053-359a875baae4\" (UID: \"2ad25116-95f1-4022-8053-359a875baae4\") " Oct 30 21:44:05.153552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153386 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.153552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153401 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfjh\" (UniqueName: \"kubernetes.io/projected/b468b352-dabe-461d-b88b-bbd0acec2ef8-kube-api-access-vjfjh\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.153552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153443 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b468b352-dabe-461d-b88b-bbd0acec2ef8-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.153552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153453 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.153552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153462 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b468b352-dabe-461d-b88b-bbd0acec2ef8-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.153797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.153693 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config" pod "2ad25116-95f1-4022-8053-359a875baae4" (UID: "2ad25116-95f1-4022-8053-359a875baae4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:05.155457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.155431 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls" pod "2ad25116-95f1-4022-8053-359a875baae4" (UID: "2ad25116-95f1-4022-8053-359a875baae4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:05.155457 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.155448 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q" pod "2ad25116-95f1-4022-8053-359a875baae4" (UID: "2ad25116-95f1-4022-8053-359a875baae4"). InnerVolumeSpecName "kube-api-access-xqq7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:05.253966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.253863 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqq7q\" (UniqueName: \"kubernetes.io/projected/2ad25116-95f1-4022-8053-359a875baae4-kube-api-access-xqq7q\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.253966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.253905 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ad25116-95f1-4022-8053-359a875baae4-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.253966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.253922 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2ad25116-95f1-4022-8053-359a875baae4-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:05.439486 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.439451 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww"] Oct 30 21:44:05.443752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.443719 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-5tgww"] Oct 30 21:44:05.947740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:05.947707 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.015895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.015817 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" path="/var/lib/kubelet/pods/b468b352-dabe-461d-b88b-bbd0acec2ef8/volumes" Oct 30 21:44:06.025342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.025318 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" Oct 30 21:44:06.025475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.025310 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5" event={"ID":"2ad25116-95f1-4022-8053-359a875baae4","Type":"ContainerDied","Data":"5230c7095a3956e05effd8e8dfd5c31ee6f1522bcb05fcf3ffaedb167dddc21f"} Oct 30 21:44:06.025522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.025483 4988 scope.go:122] "RemoveContainer" containerID="bb661a4a6f46b02efaa069347f0487be11c0627d307952bc7ca0755f51e5e21e" Oct 30 21:44:06.034029 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.034007 4988 scope.go:122] "RemoveContainer" containerID="c06e3b6765e03e2a5df3ddec14e9854f714698df02c121b40c5fcc31ce88158d" Oct 30 21:44:06.042205 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.042175 4988 scope.go:122] "RemoveContainer" containerID="2ce32181f3382ea92de6a03ecbc9d5e392134e77c3a9ebba418fed43f3645499" Oct 30 21:44:06.158087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.158042 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.196422 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.196369 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5"] Oct 30 21:44:06.202610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.202580 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-m7bt5"] Oct 30 21:44:06.750112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.750074 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.946251 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.946207 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.947104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.947078 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:44:06.947368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.947297 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="alertmanager" containerID="cri-o://0ed50966c793aab8210303e4ff6c965375f353fd50a909f333a37a49e5b86e4b" gracePeriod=120 Oct 30 21:44:06.947520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.947497 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="config-reloader" containerID="cri-o://500074f341a5cbcef6701718a484023ac74f31e228de7840a7dbbbe48a32f015" gracePeriod=120 Oct 30 21:44:06.961328 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.961298 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:44:06.961740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.961560 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="prometheus" containerID="cri-o://6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" gracePeriod=600 Oct 30 21:44:06.961740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.961578 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="config-reloader" containerID="cri-o://96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" gracePeriod=600 Oct 30 21:44:06.961740 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:06.961566 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="thanos-sidecar" containerID="cri-o://377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" gracePeriod=600 Oct 30 21:44:07.112202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.112164 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:07.217066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.217036 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:07.370331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370246 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370286 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370311 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370332 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370366 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370400 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370444 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370531 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370568 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.370654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370619 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.371020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370670 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.371020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370699 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:07.371020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370708 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.371020 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.370800 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"5888c9d7-012d-4e75-a707-71f67d648ee8\" (UID: \"5888c9d7-012d-4e75-a707-71f67d648ee8\") " Oct 30 21:44:07.371241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.371080 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.371842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.371361 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:07.373844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.373734 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:07.374744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.374704 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:07.375791 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.375761 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:07.376171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.376146 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:07.376687 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.376661 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "kube-api-access-jnxgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:07.377963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.377940 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:44:07.378725 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.378702 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:07.378807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.378722 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:07.378807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.378792 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:07.380325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.380295 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:07.391423 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.391376 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config" pod "5888c9d7-012d-4e75-a707-71f67d648ee8" (UID: "5888c9d7-012d-4e75-a707-71f67d648ee8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:07.472229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472188 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472223 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472229 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472234 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472244 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472255 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472265 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5888c9d7-012d-4e75-a707-71f67d648ee8-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472274 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472304 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472314 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472323 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxgw\" (UniqueName: \"kubernetes.io/projected/5888c9d7-012d-4e75-a707-71f67d648ee8-kube-api-access-jnxgw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472333 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5888c9d7-012d-4e75-a707-71f67d648ee8-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.472516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.472343 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5888c9d7-012d-4e75-a707-71f67d648ee8-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.484006 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.483980 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:44:07.572780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:07.572738 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:44:08.015334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.015296 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad25116-95f1-4022-8053-359a875baae4" path="/var/lib/kubelet/pods/2ad25116-95f1-4022-8053-359a875baae4/volumes" Oct 30 21:44:08.038945 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.038912 4988 generic.go:356] "Generic (PLEG): container finished" podID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerID="500074f341a5cbcef6701718a484023ac74f31e228de7840a7dbbbe48a32f015" exitCode=0 Oct 30 21:44:08.039083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.038984 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerDied","Data":"500074f341a5cbcef6701718a484023ac74f31e228de7840a7dbbbe48a32f015"} Oct 30 21:44:08.041294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041274 4988 generic.go:356] "Generic (PLEG): container finished" podID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerID="377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" exitCode=0 Oct 30 21:44:08.041294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041292 4988 generic.go:356] "Generic (PLEG): container finished" podID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerID="96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" exitCode=0 Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041299 4988 generic.go:356] "Generic (PLEG): container finished" podID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerID="6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" exitCode=0 Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041312 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerDied","Data":"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1"} Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041376 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041382 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerDied","Data":"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb"} Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041437 4988 scope.go:122] "RemoveContainer" containerID="377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041442 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerDied","Data":"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7"} Oct 30 21:44:08.041481 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.041477 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"5888c9d7-012d-4e75-a707-71f67d648ee8","Type":"ContainerDied","Data":"aff6fdfd3381fa0b9bd2d17bcc6a8b15259f958357274a06240be620e2c1b34d"} Oct 30 21:44:08.050133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.050113 4988 scope.go:122] "RemoveContainer" containerID="96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" Oct 30 21:44:08.058276 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.058248 4988 scope.go:122] "RemoveContainer" containerID="6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" Oct 30 21:44:08.066673 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.066646 4988 scope.go:122] "RemoveContainer" containerID="ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b" Oct 30 21:44:08.075211 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075009 4988 scope.go:122] "RemoveContainer" containerID="377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" Oct 30 21:44:08.075356 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:08.075272 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": container with ID starting with 377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1 not found: ID does not exist" containerID="377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" Oct 30 21:44:08.075356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075303 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1"} err="failed to get container status \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": rpc error: code = NotFound desc = could not find container \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": container with ID starting with 377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1 not found: ID does not exist" Oct 30 21:44:08.075356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075322 4988 scope.go:122] "RemoveContainer" containerID="96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" Oct 30 21:44:08.075567 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:08.075549 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": container with ID starting with 96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb not found: ID does not exist" containerID="96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" Oct 30 21:44:08.075648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075581 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb"} err="failed to get container status \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": rpc error: code = NotFound desc = could not find container \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": container with ID starting with 96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb not found: ID does not exist" Oct 30 21:44:08.075648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075611 4988 scope.go:122] "RemoveContainer" containerID="6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" Oct 30 21:44:08.075833 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:08.075816 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": container with ID starting with 6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7 not found: ID does not exist" containerID="6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" Oct 30 21:44:08.075870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075838 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7"} err="failed to get container status \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": rpc error: code = NotFound desc = could not find container \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": container with ID starting with 6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7 not found: ID does not exist" Oct 30 21:44:08.075870 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.075851 4988 scope.go:122] "RemoveContainer" containerID="ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b" Oct 30 21:44:08.076082 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:08.076062 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": container with ID starting with ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b not found: ID does not exist" containerID="ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b" Oct 30 21:44:08.076154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076089 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b"} err="failed to get container status \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": rpc error: code = NotFound desc = could not find container \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": container with ID starting with ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b not found: ID does not exist" Oct 30 21:44:08.076154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076110 4988 scope.go:122] "RemoveContainer" containerID="377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" Oct 30 21:44:08.076321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076302 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1"} err="failed to get container status \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": rpc error: code = NotFound desc = could not find container \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": container with ID starting with 377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1 not found: ID does not exist" Oct 30 21:44:08.076365 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076322 4988 scope.go:122] "RemoveContainer" containerID="96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" Oct 30 21:44:08.076560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076541 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb"} err="failed to get container status \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": rpc error: code = NotFound desc = could not find container \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": container with ID starting with 96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb not found: ID does not exist" Oct 30 21:44:08.076640 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076561 4988 scope.go:122] "RemoveContainer" containerID="6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" Oct 30 21:44:08.076771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076754 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7"} err="failed to get container status \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": rpc error: code = NotFound desc = could not find container \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": container with ID starting with 6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7 not found: ID does not exist" Oct 30 21:44:08.076840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076772 4988 scope.go:122] "RemoveContainer" containerID="ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b" Oct 30 21:44:08.076962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076944 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b"} err="failed to get container status \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": rpc error: code = NotFound desc = could not find container \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": container with ID starting with ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b not found: ID does not exist" Oct 30 21:44:08.077033 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.076963 4988 scope.go:122] "RemoveContainer" containerID="377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1" Oct 30 21:44:08.077176 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077156 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1"} err="failed to get container status \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": rpc error: code = NotFound desc = could not find container \"377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1\": container with ID starting with 377b8125d5b56e58d119fb9f8defce14385ad9f3248e4f116ebc1f048dcd34e1 not found: ID does not exist" Oct 30 21:44:08.077216 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077177 4988 scope.go:122] "RemoveContainer" containerID="96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb" Oct 30 21:44:08.077390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077374 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb"} err="failed to get container status \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": rpc error: code = NotFound desc = could not find container \"96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb\": container with ID starting with 96d6169d33e746326ab7986be290b4d1934133451ce0d452df16e05d870a60bb not found: ID does not exist" Oct 30 21:44:08.077467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077391 4988 scope.go:122] "RemoveContainer" containerID="6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7" Oct 30 21:44:08.077717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077697 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7"} err="failed to get container status \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": rpc error: code = NotFound desc = could not find container \"6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7\": container with ID starting with 6a630ac7e70bc6db33a9b3886d476c956ac47dfe9c94f2e3b5c0aa317ed647b7 not found: ID does not exist" Oct 30 21:44:08.077770 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077717 4988 scope.go:122] "RemoveContainer" containerID="ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b" Oct 30 21:44:08.077928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.077909 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b"} err="failed to get container status \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": rpc error: code = NotFound desc = could not find container \"ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b\": container with ID starting with ae0705631345959f73239a61e2ac92eb649f6a3fbc3480a62755f3e279cde13b not found: ID does not exist" Oct 30 21:44:08.219592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.219552 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:44:08.223124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.223096 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:44:08.542567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.542532 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:44:08.542781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.542720 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0fd73a97-1a06-45c7-b250-3ca9c7579e36" containerName="otc-container" containerID="cri-o://4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab" gracePeriod=30 Oct 30 21:44:08.596756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.596715 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.141:9093/-/ready\": dial tcp 10.132.0.141:9093: connect: connection refused" Oct 30 21:44:08.604752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.604715 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:08.763486 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.763448 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:08.798635 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.798570 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:08.985058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985024 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs\") pod \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " Oct 30 21:44:08.985267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985079 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert\") pod \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " Oct 30 21:44:08.985267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985110 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4\" (UniqueName: \"kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4\") pod \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " Oct 30 21:44:08.985482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985301 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal\") pod \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " Oct 30 21:44:08.985482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985345 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs\") pod \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " Oct 30 21:44:08.985482 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985404 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs\") pod \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\" (UID: \"0fd73a97-1a06-45c7-b250-3ca9c7579e36\") " Oct 30 21:44:08.985732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.985701 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal" pod "0fd73a97-1a06-45c7-b250-3ca9c7579e36" (UID: "0fd73a97-1a06-45c7-b250-3ca9c7579e36"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:08.987812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.987777 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs" pod "0fd73a97-1a06-45c7-b250-3ca9c7579e36" (UID: "0fd73a97-1a06-45c7-b250-3ca9c7579e36"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:08.987812 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.987791 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4" pod "0fd73a97-1a06-45c7-b250-3ca9c7579e36" (UID: "0fd73a97-1a06-45c7-b250-3ca9c7579e36"). InnerVolumeSpecName "kube-api-access-vc9b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:08.988011 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.987827 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs" pod "0fd73a97-1a06-45c7-b250-3ca9c7579e36" (UID: "0fd73a97-1a06-45c7-b250-3ca9c7579e36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:08.988011 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.987859 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert" pod "0fd73a97-1a06-45c7-b250-3ca9c7579e36" (UID: "0fd73a97-1a06-45c7-b250-3ca9c7579e36"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:08.988119 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:08.988016 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs" pod "0fd73a97-1a06-45c7-b250-3ca9c7579e36" (UID: "0fd73a97-1a06-45c7-b250-3ca9c7579e36"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:09.022133 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.022096 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:09.048246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.048207 4988 generic.go:356] "Generic (PLEG): container finished" podID="0fd73a97-1a06-45c7-b250-3ca9c7579e36" containerID="4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab" exitCode=0 Oct 30 21:44:09.048391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.048250 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:09.048391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.048252 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0fd73a97-1a06-45c7-b250-3ca9c7579e36","Type":"ContainerDied","Data":"4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab"} Oct 30 21:44:09.048391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.048381 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0fd73a97-1a06-45c7-b250-3ca9c7579e36","Type":"ContainerDied","Data":"19266e3448cdfc1c6bc9813089498a3a324440646726e976d59a7306a1ae37b2"} Oct 30 21:44:09.048560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.048405 4988 scope.go:122] "RemoveContainer" containerID="4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab" Oct 30 21:44:09.060731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.060701 4988 scope.go:122] "RemoveContainer" containerID="4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab" Oct 30 21:44:09.061296 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:09.061265 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab\": container with ID starting with 4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab not found: ID does not exist" containerID="4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab" Oct 30 21:44:09.061492 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.061309 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab"} err="failed to get container status \"4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab\": rpc error: code = NotFound desc = could not find container \"4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab\": container with ID starting with 4cf488f923a6def92c902788b5af08542e199b10e994da1ac258cb27a045e6ab not found: ID does not exist" Oct 30 21:44:09.086943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.086910 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:09.086943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.086944 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:09.087146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.086958 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:09.087146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.086971 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:09.087146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.086985 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0fd73a97-1a06-45c7-b250-3ca9c7579e36-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:09.087146 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.086999 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9b4\" (UniqueName: \"kubernetes.io/projected/0fd73a97-1a06-45c7-b250-3ca9c7579e36-kube-api-access-vc9b4\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:09.231689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.231655 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:44:09.236290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.236262 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:44:09.474686 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:09.474649 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.016051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:10.016016 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd73a97-1a06-45c7-b250-3ca9c7579e36" path="/var/lib/kubelet/pods/0fd73a97-1a06-45c7-b250-3ca9c7579e36/volumes" Oct 30 21:44:10.016518 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:10.016503 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" path="/var/lib/kubelet/pods/5888c9d7-012d-4e75-a707-71f67d648ee8/volumes" Oct 30 21:44:10.250256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:10.250220 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.453517 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:10.453482 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.620018 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:10.619977 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:10.775102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:10.775002 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:12.836764 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:12.836732 4988 scope.go:122] "RemoveContainer" containerID="67910180fd20a7949b8a69693a3d0ecf6bef1bfdce42addfc4023284beffe394" Oct 30 21:44:13.597446 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:13.597371 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.141:9093/-/ready\": dial tcp 10.132.0.141:9093: connect: connection refused" Oct 30 21:44:17.087452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.087394 4988 generic.go:356] "Generic (PLEG): container finished" podID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerID="0ed50966c793aab8210303e4ff6c965375f353fd50a909f333a37a49e5b86e4b" exitCode=0 Oct 30 21:44:17.087835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.087452 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerDied","Data":"0ed50966c793aab8210303e4ff6c965375f353fd50a909f333a37a49e5b86e4b"} Oct 30 21:44:17.208794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.208726 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:17.252846 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.252814 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.252990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.252862 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.252990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.252888 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.252990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.252936 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.252990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.252953 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.252990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.252983 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.253273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.253018 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db\") pod \"4b020adf-0e54-4e96-a67d-56a593f1137d\" (UID: \"4b020adf-0e54-4e96-a67d-56a593f1137d\") " Oct 30 21:44:17.253485 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.253454 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:17.255252 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.255223 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:17.255351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.255272 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:17.255546 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.255525 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "kube-api-access-864lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:17.255632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.255607 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:17.259357 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.259333 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:17.267325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.267287 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config" pod "4b020adf-0e54-4e96-a67d-56a593f1137d" (UID: "4b020adf-0e54-4e96-a67d-56a593f1137d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:17.354042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354007 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.354042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354037 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.354042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354047 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.354288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354058 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4b020adf-0e54-4e96-a67d-56a593f1137d-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.354288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354067 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864lx\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-kube-api-access-864lx\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.354288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354077 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b020adf-0e54-4e96-a67d-56a593f1137d-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:17.354288 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:17.354085 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b020adf-0e54-4e96-a67d-56a593f1137d-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:18.093531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.093489 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4b020adf-0e54-4e96-a67d-56a593f1137d","Type":"ContainerDied","Data":"87a0ec1965b7719c881cf2d67037091238d2a35d5868ee8db3e63b965c7357c4"} Oct 30 21:44:18.094088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.093561 4988 scope.go:122] "RemoveContainer" containerID="500074f341a5cbcef6701718a484023ac74f31e228de7840a7dbbbe48a32f015" Oct 30 21:44:18.094088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.093500 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:18.101905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.101878 4988 scope.go:122] "RemoveContainer" containerID="0ed50966c793aab8210303e4ff6c965375f353fd50a909f333a37a49e5b86e4b" Oct 30 21:44:18.109450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.109429 4988 scope.go:122] "RemoveContainer" containerID="33f973591d3cb9cdc86039e1bfe8f4573f506419bf1d4e5eef36ff903b086346" Oct 30 21:44:18.323940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.323847 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:44:18.373185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:18.373151 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:44:20.015090 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:20.015053 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" path="/var/lib/kubelet/pods/4b020adf-0e54-4e96-a67d-56a593f1137d/volumes" Oct 30 21:44:24.786359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:24.786277 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.490823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:25.490783 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.744541 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:25.744447 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.949002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:25.948967 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.160253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.160217 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.364050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.364011 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.393397 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:26.393362 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:44:26.565466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565365 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4"] Oct 30 21:44:26.565794 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565775 4988 cpu_manager.go:414] "containerMap: removing container" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="init-config-reloader" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565796 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="init-config-reloader" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565813 4988 cpu_manager.go:414] "containerMap: removing container" podUID="0fd73a97-1a06-45c7-b250-3ca9c7579e36" containerName="otc-container" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565819 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd73a97-1a06-45c7-b250-3ca9c7579e36" containerName="otc-container" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565834 4988 cpu_manager.go:414] "containerMap: removing container" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="kube-rbac-proxy" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565842 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="kube-rbac-proxy" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565861 4988 cpu_manager.go:414] "containerMap: removing container" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="method-gate" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565870 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="method-gate" Oct 30 21:44:26.565881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565880 4988 cpu_manager.go:414] "containerMap: removing container" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="prom-label-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565887 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="prom-label-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565896 4988 cpu_manager.go:414] "containerMap: removing container" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="thanos-sidecar" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565904 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="thanos-sidecar" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565916 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="prom-label-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565924 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="prom-label-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565939 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="kube-rbac-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565947 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="kube-rbac-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565957 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="alertmanager" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565964 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="alertmanager" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565975 4988 cpu_manager.go:414] "containerMap: removing container" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="prometheus" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565982 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="prometheus" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.565995 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="init-config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566002 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="init-config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566015 4988 cpu_manager.go:414] "containerMap: removing container" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="method-gate" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566024 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="method-gate" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566037 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566044 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566060 4988 cpu_manager.go:414] "containerMap: removing container" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566067 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566153 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="method-gate" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566167 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="prom-label-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566176 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="thanos-sidecar" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566186 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="prom-label-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566194 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="kube-rbac-proxy" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566202 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="alertmanager" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566210 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="prometheus" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566219 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad25116-95f1-4022-8053-359a875baae4" containerName="method-gate" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566227 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b020adf-0e54-4e96-a67d-56a593f1137d" containerName="config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566241 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5888c9d7-012d-4e75-a707-71f67d648ee8" containerName="config-reloader" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566252 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd73a97-1a06-45c7-b250-3ca9c7579e36" containerName="otc-container" Oct 30 21:44:26.566351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.566260 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b468b352-dabe-461d-b88b-bbd0acec2ef8" containerName="kube-rbac-proxy" Oct 30 21:44:26.570126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.570105 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.604026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.603994 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.633692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.633659 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.633905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.633710 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5f8j\" (UniqueName: \"kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.633905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.633786 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.633905 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.633880 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.634080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.633926 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.678320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678273 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:44:26.678320 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678313 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:44:26.678591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678327 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-pmqmm\"" Oct 30 21:44:26.678591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678338 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:44:26.678591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678364 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:44:26.678591 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678364 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:44:26.678800 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.678784 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:44:26.680160 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.680132 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4"] Oct 30 21:44:26.730389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.730349 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd"] Oct 30 21:44:26.733941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.733920 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.734365 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.734336 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.734480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.734378 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.734480 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.734447 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.734606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.734486 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5f8j\" (UniqueName: \"kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.734606 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.734510 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.735064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.735041 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.735132 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.735116 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.735183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.735126 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.736895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.736877 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.781191 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.781146 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd"] Oct 30 21:44:26.791337 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.791308 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-zdk8w\"" Oct 30 21:44:26.791514 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.791369 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:44:26.793169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.793148 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:44:26.800739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.800708 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5f8j\" (UniqueName: \"kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.814081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.814044 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:26.835717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.835608 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.835881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.835736 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.835881 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.835772 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzxn\" (UniqueName: \"kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.887588 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.887541 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:44:26.936844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.936800 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.937018 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.936895 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.937018 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.936927 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzxn\" (UniqueName: \"kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.940461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.940435 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.940645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.940558 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:26.999244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:26.999205 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzxn\" (UniqueName: \"kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn\") pod \"data-science-thanos-querier-proxy-77657866bd-rq5xd\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:27.058990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:27.058953 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:44:27.270909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:27.270878 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4"] Oct 30 21:44:27.274054 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:44:27.274024 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6878e97_4390_4316_bd13_0675fc01b206.slice/crio-e973027a5dda90c45ee349a3c8083d876b800c2c31f5e7aa2fe61305d2cbe1ce WatchSource:0}: Error finding container e973027a5dda90c45ee349a3c8083d876b800c2c31f5e7aa2fe61305d2cbe1ce: Status 404 returned error can't find the container with id e973027a5dda90c45ee349a3c8083d876b800c2c31f5e7aa2fe61305d2cbe1ce Oct 30 21:44:27.454233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:27.454203 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd"] Oct 30 21:44:27.457883 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:44:27.457850 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cffdbb_da2b_435f_83c1_d4fa16f3ea1a.slice/crio-17a9391c6c33cb08a6d1431c6a9c890e8aac4b7e8cebb75a9c61b276f9cc4162 WatchSource:0}: Error finding container 17a9391c6c33cb08a6d1431c6a9c890e8aac4b7e8cebb75a9c61b276f9cc4162: Status 404 returned error can't find the container with id 17a9391c6c33cb08a6d1431c6a9c890e8aac4b7e8cebb75a9c61b276f9cc4162 Oct 30 21:44:28.138990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.138958 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerStarted","Data":"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8"} Oct 30 21:44:28.139467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.139017 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerStarted","Data":"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24"} Oct 30 21:44:28.139467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.139047 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerStarted","Data":"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f"} Oct 30 21:44:28.139467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.139074 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerStarted","Data":"17a9391c6c33cb08a6d1431c6a9c890e8aac4b7e8cebb75a9c61b276f9cc4162"} Oct 30 21:44:28.141048 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.141027 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerStarted","Data":"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569"} Oct 30 21:44:28.141134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.141072 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerStarted","Data":"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c"} Oct 30 21:44:28.141134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.141102 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerStarted","Data":"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852"} Oct 30 21:44:28.141134 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.141125 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerStarted","Data":"e973027a5dda90c45ee349a3c8083d876b800c2c31f5e7aa2fe61305d2cbe1ce"} Oct 30 21:44:28.389912 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.389809 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" podStartSLOduration=2.3897936140000002 podStartE2EDuration="2.389793614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:28.265677242 +0000 UTC m=+4096.373173154" watchObservedRunningTime="2026-10-30 21:44:28.389793614 +0000 UTC m=+4096.497289553" Oct 30 21:44:28.390441 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.390395 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" podStartSLOduration=2.390389233 podStartE2EDuration="2.390389233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:28.388535486 +0000 UTC m=+4096.496031397" watchObservedRunningTime="2026-10-30 21:44:28.390389233 +0000 UTC m=+4096.497885139" Oct 30 21:44:28.401137 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:28.401091 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:31.557679 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.557647 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:31.584989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.584948 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:44:31.588559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.588534 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.666714 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.666676 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:44:31.678467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.678434 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.678693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.678500 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgct\" (UniqueName: \"kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.678693 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.678660 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.678852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.678774 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.678852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.678811 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.678852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.678844 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.703150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.703108 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:44:31.714147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.714110 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:44:31.714147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.714109 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:44:31.714147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.714118 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7bnjl\"" Oct 30 21:44:31.714147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.714120 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:44:31.714448 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.714226 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:44:31.738273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.738232 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:31.779611 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.779568 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.779611 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.779618 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.779887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.779648 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.779887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.779728 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.779887 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.779780 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgct\" (UniqueName: \"kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.780065 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.779972 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.780894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.780861 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.782476 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.782446 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.782476 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.782473 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.782623 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.782479 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.782662 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.782613 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.841756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.841683 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgct\" (UniqueName: \"kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct\") pod \"data-science-collector-collector-1\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.902504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.902468 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:44:31.916404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:31.916369 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:32.093263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:32.093165 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:32.294479 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:32.294439 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:44:32.297540 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:44:32.297516 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fbf2c0_c3c4_431c_b497_031d69e5cfd0.slice/crio-87ff23a6110e069026246e4b2b82f26805fd25263f64dfc3f8d1bfd0f139acff WatchSource:0}: Error finding container 87ff23a6110e069026246e4b2b82f26805fd25263f64dfc3f8d1bfd0f139acff: Status 404 returned error can't find the container with id 87ff23a6110e069026246e4b2b82f26805fd25263f64dfc3f8d1bfd0f139acff Oct 30 21:44:32.299843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:32.299805 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:32.299963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:32.299890 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:33.165645 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:33.165607 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"00fbf2c0-c3c4-431c-b497-031d69e5cfd0","Type":"ContainerStarted","Data":"4d081572adebd1ff0977c32f903a7e5db197171d46c6bd3336afd8d668ceadd7"} Oct 30 21:44:33.166209 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:33.165673 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"00fbf2c0-c3c4-431c-b497-031d69e5cfd0","Type":"ContainerStarted","Data":"87ff23a6110e069026246e4b2b82f26805fd25263f64dfc3f8d1bfd0f139acff"} Oct 30 21:44:33.289501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:33.289448 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.289428359 podStartE2EDuration="2.289428359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:33.289018456 +0000 UTC m=+4101.396514368" watchObservedRunningTime="2026-10-30 21:44:33.289428359 +0000 UTC m=+4101.396924268" Oct 30 21:44:44.573727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:44.573688 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:44.875461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:44.875343 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.060713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:45.060675 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.269710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:45.269674 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.792400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:45.792357 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:46.032710 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.032674 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:44:46.035107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.035060 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:46.037391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.037372 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.062695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.062659 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:44:46.066524 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.066496 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.098542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.098508 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:44:46.110703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.110672 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:44:46.113436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.113368 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:44:46.152330 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.152301 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:44:46.199190 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199153 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:44:46.199400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199340 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:44:46.199400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199372 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:44:46.199400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199372 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199433 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199471 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fq8lz\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199456 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199457 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199489 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199459 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199513 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:44:46.199625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.199459 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:44:46.200325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.200312 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-lr9wm\"" Oct 30 21:44:46.208310 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208287 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208441 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208320 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208441 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208341 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208500 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208525 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208570 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208559 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208643 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208582 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208643 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208631 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxfx\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208713 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208699 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208725 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208742 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.208824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.208764 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.309595 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309488 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.309595 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309548 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309603 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309643 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309661 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309709 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309738 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309758 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309778 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgpn\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.309871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309854 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309904 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309945 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxfx\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.309988 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.310301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310014 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.310301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310046 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310076 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.310670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310440 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310496 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310670 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310581 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.310844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.310783 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.311467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.311434 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.311619 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.311520 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.313124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.313093 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.313784 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.313546 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.313880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.313616 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.313880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.313633 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.313880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.313701 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.313880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.313727 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.314068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.314051 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.314389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.314367 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.383355 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.383317 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxfx\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:46.411503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411464 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.411503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411502 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.411788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411520 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.411788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411553 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.411788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411698 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.411788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411742 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.412031 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.411795 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgpn\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.412031 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.412001 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.414247 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.414215 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.414247 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.414227 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.414449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.414213 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.414509 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.414492 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.414550 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.414517 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.473979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.473925 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgpn\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:46.684406 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:46.684366 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:47.121358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:47.121278 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:44:47.124124 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:44:47.124096 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ab3b5f_62ac_49c5_a70a_d201b57d9fe0.slice/crio-a135d857d1dedb1181af555ed678db8ced09f32ddd58e26dddcbf883aecb5a68 WatchSource:0}: Error finding container a135d857d1dedb1181af555ed678db8ced09f32ddd58e26dddcbf883aecb5a68: Status 404 returned error can't find the container with id a135d857d1dedb1181af555ed678db8ced09f32ddd58e26dddcbf883aecb5a68 Oct 30 21:44:47.234963 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:47.234927 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerStarted","Data":"a135d857d1dedb1181af555ed678db8ced09f32ddd58e26dddcbf883aecb5a68"} Oct 30 21:44:48.240960 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:48.240922 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerStarted","Data":"3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40"} Oct 30 21:44:48.835731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:48.835688 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:48.954517 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:48.954479 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.037610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.037574 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.037808 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.037706 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.089399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.089320 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.227865 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.227832 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.295512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.295481 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.358938 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.358832 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:44:49.846396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:49.846356 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:44:49.889708 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:44:49.889677 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbabd88_a421_4c8f_adfb_3c541f515568.slice/crio-26633a9f57c4e5c0cff4326a9f4311b44702c3e2b28b5909f65c5184c2b017c7 WatchSource:0}: Error finding container 26633a9f57c4e5c0cff4326a9f4311b44702c3e2b28b5909f65c5184c2b017c7: Status 404 returned error can't find the container with id 26633a9f57c4e5c0cff4326a9f4311b44702c3e2b28b5909f65c5184c2b017c7 Oct 30 21:44:50.250544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:50.250508 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerStarted","Data":"26633a9f57c4e5c0cff4326a9f4311b44702c3e2b28b5909f65c5184c2b017c7"} Oct 30 21:44:51.256440 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:51.256387 4988 generic.go:356] "Generic (PLEG): container finished" podID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerID="3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40" exitCode=0 Oct 30 21:44:51.256892 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:51.256458 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerDied","Data":"3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40"} Oct 30 21:44:51.258155 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:51.258127 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerStarted","Data":"ced10f250228ba1f402b4327ef877767ce8daab918e83b4b9abb891cc40a2ec3"} Oct 30 21:44:52.265488 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:52.265455 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerStarted","Data":"1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e"} Oct 30 21:44:52.265854 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:52.265516 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerStarted","Data":"9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430"} Oct 30 21:44:52.265923 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:52.265880 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:52.394123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:52.394060 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.908521145 podStartE2EDuration="6.39404575s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.485524605s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:52.392895095 +0000 UTC m=+4120.500391006" watchObservedRunningTime="2026-10-30 21:44:52.39404575 +0000 UTC m=+4120.501541700" Oct 30 21:44:54.275654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:54.275538 4988 generic.go:356] "Generic (PLEG): container finished" podID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerID="ced10f250228ba1f402b4327ef877767ce8daab918e83b4b9abb891cc40a2ec3" exitCode=0 Oct 30 21:44:54.275654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:54.275632 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerDied","Data":"ced10f250228ba1f402b4327ef877767ce8daab918e83b4b9abb891cc40a2ec3"} Oct 30 21:44:55.285404 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:55.285362 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerStarted","Data":"ec6a1131c6e99546cd29e3a5f2daf0909a3639d2c22b4739d85b555ee93fb280"} Oct 30 21:44:55.285823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:55.285497 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerStarted","Data":"997d5ebbf294c5092ef514f2d9e66948a849d293fc439831b736fd1e3ddff5c8"} Oct 30 21:44:55.285823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:55.285545 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerStarted","Data":"1a5dcd1d3abdc5d5265b37089ac20b74fdd2dd192c56c9cdfb9b7bdd5fce9a54"} Oct 30 21:44:55.421370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:55.421314 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.935065125 podStartE2EDuration="10.421302043s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.486236918s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:55.420725687 +0000 UTC m=+4123.528221598" watchObservedRunningTime="2026-10-30 21:44:55.421302043 +0000 UTC m=+4123.528797953" Oct 30 21:44:56.394680 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:44:56.394641 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:44:58.271917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:58.271882 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:44:59.359490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:44:59.359450 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:45:04.359235 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:04.359193 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:45:04.364738 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:04.364713 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:45:05.335075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:05.335044 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:45:12.367849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:12.367812 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:12.515920 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:12.515880 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:12.717551 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:12.717510 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:12.920066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:12.920024 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:13.064135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:13.064012 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:13.211624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:13.211572 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.146577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:18.146536 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:18.147084 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:18.146734 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="00fbf2c0-c3c4-431c-b497-031d69e5cfd0" containerName="otc-container" containerID="cri-o://4d081572adebd1ff0977c32f903a7e5db197171d46c6bd3336afd8d668ceadd7" gracePeriod=30 Oct 30 21:45:18.523965 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:18.523862 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.716266 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:18.716229 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:18.904688 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:18.904648 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:20.402253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.402215 4988 generic.go:356] "Generic (PLEG): container finished" podID="00fbf2c0-c3c4-431c-b497-031d69e5cfd0" containerID="4d081572adebd1ff0977c32f903a7e5db197171d46c6bd3336afd8d668ceadd7" exitCode=0 Oct 30 21:45:20.402683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.402293 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"00fbf2c0-c3c4-431c-b497-031d69e5cfd0","Type":"ContainerDied","Data":"4d081572adebd1ff0977c32f903a7e5db197171d46c6bd3336afd8d668ceadd7"} Oct 30 21:45:20.492646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.492611 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:20.502655 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.502626 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs\") pod \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " Oct 30 21:45:20.502781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.502670 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert\") pod \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " Oct 30 21:45:20.502781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.502695 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs\") pod \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " Oct 30 21:45:20.502781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.502736 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct\" (UniqueName: \"kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct\") pod \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " Oct 30 21:45:20.502781 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.502770 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal\") pod \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " Oct 30 21:45:20.502997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.502835 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs\") pod \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\" (UID: \"00fbf2c0-c3c4-431c-b497-031d69e5cfd0\") " Oct 30 21:45:20.503185 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.503148 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal" pod "00fbf2c0-c3c4-431c-b497-031d69e5cfd0" (UID: "00fbf2c0-c3c4-431c-b497-031d69e5cfd0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.504985 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.504958 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct" pod "00fbf2c0-c3c4-431c-b497-031d69e5cfd0" (UID: "00fbf2c0-c3c4-431c-b497-031d69e5cfd0"). InnerVolumeSpecName "kube-api-access-xpgct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.505091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.505038 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert" pod "00fbf2c0-c3c4-431c-b497-031d69e5cfd0" (UID: "00fbf2c0-c3c4-431c-b497-031d69e5cfd0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:20.505219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.505202 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs" pod "00fbf2c0-c3c4-431c-b497-031d69e5cfd0" (UID: "00fbf2c0-c3c4-431c-b497-031d69e5cfd0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:20.505423 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.505383 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs" pod "00fbf2c0-c3c4-431c-b497-031d69e5cfd0" (UID: "00fbf2c0-c3c4-431c-b497-031d69e5cfd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:20.505520 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.505431 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs" pod "00fbf2c0-c3c4-431c-b497-031d69e5cfd0" (UID: "00fbf2c0-c3c4-431c-b497-031d69e5cfd0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:20.603837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.603757 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.603837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.603786 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.603837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.603797 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpgct\" (UniqueName: \"kubernetes.io/projected/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-kube-api-access-xpgct\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.603837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.603807 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.603837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.603819 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.603837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:20.603829 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fbf2c0-c3c4-431c-b497-031d69e5cfd0-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.408129 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.408096 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.408583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.408097 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"00fbf2c0-c3c4-431c-b497-031d69e5cfd0","Type":"ContainerDied","Data":"87ff23a6110e069026246e4b2b82f26805fd25263f64dfc3f8d1bfd0f139acff"} Oct 30 21:45:21.408583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.408238 4988 scope.go:122] "RemoveContainer" containerID="4d081572adebd1ff0977c32f903a7e5db197171d46c6bd3336afd8d668ceadd7" Oct 30 21:45:21.578883 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.578835 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:21.591983 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.591944 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:21.703098 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.703021 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:21.703436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.703403 4988 cpu_manager.go:414] "containerMap: removing container" podUID="00fbf2c0-c3c4-431c-b497-031d69e5cfd0" containerName="otc-container" Oct 30 21:45:21.703436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.703430 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fbf2c0-c3c4-431c-b497-031d69e5cfd0" containerName="otc-container" Oct 30 21:45:21.703533 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.703516 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fbf2c0-c3c4-431c-b497-031d69e5cfd0" containerName="otc-container" Oct 30 21:45:21.707882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.707863 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.711106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.711078 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.711241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.711119 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.711241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.711234 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkpt\" (UniqueName: \"kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.711383 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.711273 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.711383 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.711337 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.711496 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.711434 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.773429 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.773389 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:45:21.773750 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.773727 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:45:21.773814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.773727 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7bnjl\"" Oct 30 21:45:21.773814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.773733 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:45:21.773814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.773784 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:45:21.773928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.773842 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:45:21.812072 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.812030 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkpt\" (UniqueName: \"kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.812072 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.812075 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.812296 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.812133 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.812351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.812333 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.812471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.812453 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.812527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.812486 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.813106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.813081 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.814755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.814727 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.814871 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.814851 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.815088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.815072 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.815128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.815111 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:21.822488 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.822465 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:21.893907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:21.893867 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkpt\" (UniqueName: \"kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt\") pod \"data-science-collector-collector-1\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:22.015807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:22.015728 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fbf2c0-c3c4-431c-b497-031d69e5cfd0" path="/var/lib/kubelet/pods/00fbf2c0-c3c4-431c-b497-031d69e5cfd0/volumes" Oct 30 21:45:22.020837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:22.020788 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:22.415579 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:22.415542 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:22.419176 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:45:22.419148 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b89e20_91df_45ab_a181_6abe18003255.slice/crio-1c11187a3ea38b528c39eb8f484aadbcb1bfb07a5a3ba3dd3d865f79cfb7448a WatchSource:0}: Error finding container 1c11187a3ea38b528c39eb8f484aadbcb1bfb07a5a3ba3dd3d865f79cfb7448a: Status 404 returned error can't find the container with id 1c11187a3ea38b528c39eb8f484aadbcb1bfb07a5a3ba3dd3d865f79cfb7448a Oct 30 21:45:22.421084 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:22.421050 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:22.421188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:22.421118 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:23.420934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:23.420893 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19b89e20-91df-45ab-a181-6abe18003255","Type":"ContainerStarted","Data":"18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4"} Oct 30 21:45:23.421350 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:23.420982 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19b89e20-91df-45ab-a181-6abe18003255","Type":"ContainerStarted","Data":"1c11187a3ea38b528c39eb8f484aadbcb1bfb07a5a3ba3dd3d865f79cfb7448a"} Oct 30 21:45:23.592150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:23.592080 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.592062146 podStartE2EDuration="2.592062146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:23.541943455 +0000 UTC m=+4151.649439370" watchObservedRunningTime="2026-10-30 21:45:23.592062146 +0000 UTC m=+4151.699558059" Oct 30 21:45:25.122252 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:25.122208 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:26.395063 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:45:26.395020 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:45:26.480874 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:26.480841 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:44.285246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:44.285205 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:44.489677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:44.489646 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:44.639995 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:44.639965 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:44.785213 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:44.785174 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:44.933114 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:44.933028 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:45.094299 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:45.094260 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:50.062253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:50.062215 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:50.062823 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:50.062436 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="19b89e20-91df-45ab-a181-6abe18003255" containerName="otc-container" containerID="cri-o://18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4" gracePeriod=30 Oct 30 21:45:50.524778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:50.524732 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:50.701026 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:50.700985 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:50.850331 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:50.850222 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:52.417392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.417360 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.494290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494194 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs\") pod \"19b89e20-91df-45ab-a181-6abe18003255\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " Oct 30 21:45:52.494290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494238 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs\") pod \"19b89e20-91df-45ab-a181-6abe18003255\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " Oct 30 21:45:52.494290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494267 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs\") pod \"19b89e20-91df-45ab-a181-6abe18003255\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " Oct 30 21:45:52.494625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494299 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt\" (UniqueName: \"kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt\") pod \"19b89e20-91df-45ab-a181-6abe18003255\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " Oct 30 21:45:52.494625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494348 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal\" (UniqueName: \"kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal\") pod \"19b89e20-91df-45ab-a181-6abe18003255\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " Oct 30 21:45:52.494625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494444 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert\") pod \"19b89e20-91df-45ab-a181-6abe18003255\" (UID: \"19b89e20-91df-45ab-a181-6abe18003255\") " Oct 30 21:45:52.494856 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.494824 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal" pod "19b89e20-91df-45ab-a181-6abe18003255" (UID: "19b89e20-91df-45ab-a181-6abe18003255"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:52.496643 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.496611 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs" pod "19b89e20-91df-45ab-a181-6abe18003255" (UID: "19b89e20-91df-45ab-a181-6abe18003255"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:52.496787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.496680 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert" pod "19b89e20-91df-45ab-a181-6abe18003255" (UID: "19b89e20-91df-45ab-a181-6abe18003255"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:52.496787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.496702 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs" pod "19b89e20-91df-45ab-a181-6abe18003255" (UID: "19b89e20-91df-45ab-a181-6abe18003255"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:52.496980 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.496962 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt" pod "19b89e20-91df-45ab-a181-6abe18003255" (UID: "19b89e20-91df-45ab-a181-6abe18003255"). InnerVolumeSpecName "kube-api-access-kgkpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:52.497022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.496959 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs" pod "19b89e20-91df-45ab-a181-6abe18003255" (UID: "19b89e20-91df-45ab-a181-6abe18003255"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:52.557814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.557781 4988 generic.go:356] "Generic (PLEG): container finished" podID="19b89e20-91df-45ab-a181-6abe18003255" containerID="18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4" exitCode=0 Oct 30 21:45:52.558002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.557840 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.558002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.557871 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19b89e20-91df-45ab-a181-6abe18003255","Type":"ContainerDied","Data":"18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4"} Oct 30 21:45:52.558002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.557949 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"19b89e20-91df-45ab-a181-6abe18003255","Type":"ContainerDied","Data":"1c11187a3ea38b528c39eb8f484aadbcb1bfb07a5a3ba3dd3d865f79cfb7448a"} Oct 30 21:45:52.558002 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.557987 4988 scope.go:122] "RemoveContainer" containerID="18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4" Oct 30 21:45:52.569537 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.569509 4988 scope.go:122] "RemoveContainer" containerID="18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4" Oct 30 21:45:52.569972 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:45:52.569941 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4\": container with ID starting with 18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4 not found: ID does not exist" containerID="18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4" Oct 30 21:45:52.570076 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.569990 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4"} err="failed to get container status \"18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4\": rpc error: code = NotFound desc = could not find container \"18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4\": container with ID starting with 18407307c9ef847ccc3a8a29f18cc86d47158ea5476d7eb60126ef0c8585e8d4 not found: ID does not exist" Oct 30 21:45:52.595941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.595892 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:52.595941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.595932 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:52.595941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.595942 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:52.596194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.595953 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkpt\" (UniqueName: \"kubernetes.io/projected/19b89e20-91df-45ab-a181-6abe18003255-kube-api-access-kgkpt\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:52.596194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.595964 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/19b89e20-91df-45ab-a181-6abe18003255-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:52.596194 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.595974 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/19b89e20-91df-45ab-a181-6abe18003255-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:52.727218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.727179 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:52.732636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.732607 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:52.812785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.812694 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:52.813372 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.813351 4988 cpu_manager.go:414] "containerMap: removing container" podUID="19b89e20-91df-45ab-a181-6abe18003255" containerName="otc-container" Oct 30 21:45:52.813436 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.813376 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b89e20-91df-45ab-a181-6abe18003255" containerName="otc-container" Oct 30 21:45:52.813515 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.813502 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b89e20-91df-45ab-a181-6abe18003255" containerName="otc-container" Oct 30 21:45:52.818112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.818092 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.875644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.875608 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:45:52.875644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.875620 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:45:52.875894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.875707 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:45:52.875894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.875750 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:45:52.876554 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.876529 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7bnjl\"" Oct 30 21:45:52.876726 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.876559 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:45:52.877534 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.877509 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:52.898761 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.898728 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.898991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.898778 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fsw\" (UniqueName: \"kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.898991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.898861 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.898991 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.898972 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.899170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.899016 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:52.899170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:52.899051 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.000523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.000474 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.000523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.000522 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.000816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.000543 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.000816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.000579 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.000816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.000610 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fsw\" (UniqueName: \"kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.000816 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.000626 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.001351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.001322 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.003126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.003098 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.003280 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.003254 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.003506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.003489 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.003556 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.003489 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.063128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.063049 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fsw\" (UniqueName: \"kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw\") pod \"data-science-collector-collector-1\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.131080 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.131041 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:45:53.512121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.512060 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:45:53.514911 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:45:53.514882 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761906b9_f754_4d00_8a8d_4df9d5c80777.slice/crio-f13d7958b83a84496f8796a4ec52c263bca50cdc45e2d8b1f586c4e8deca654c WatchSource:0}: Error finding container f13d7958b83a84496f8796a4ec52c263bca50cdc45e2d8b1f586c4e8deca654c: Status 404 returned error can't find the container with id f13d7958b83a84496f8796a4ec52c263bca50cdc45e2d8b1f586c4e8deca654c Oct 30 21:45:53.517147 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.517113 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:53.517212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.517182 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:53.564954 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:53.564923 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"761906b9-f754-4d00-8a8d-4df9d5c80777","Type":"ContainerStarted","Data":"f13d7958b83a84496f8796a4ec52c263bca50cdc45e2d8b1f586c4e8deca654c"} Oct 30 21:45:54.015806 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:54.015768 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b89e20-91df-45ab-a181-6abe18003255" path="/var/lib/kubelet/pods/19b89e20-91df-45ab-a181-6abe18003255/volumes" Oct 30 21:45:54.570804 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:45:54.570771 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"761906b9-f754-4d00-8a8d-4df9d5c80777","Type":"ContainerStarted","Data":"0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001"} Oct 30 21:45:56.395359 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:45:56.395317 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:46:07.949831 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:07.949759 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=15.949743313 podStartE2EDuration="15.949743313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:54.693890675 +0000 UTC m=+4182.801386587" watchObservedRunningTime="2026-10-30 21:46:07.949743313 +0000 UTC m=+4196.057239224" Oct 30 21:46:07.950955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:07.950935 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4"] Oct 30 21:46:07.951184 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:07.951165 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="kube-rbac-proxy" containerID="cri-o://5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" gracePeriod=30 Oct 30 21:46:07.951268 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:07.951228 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="prom-label-proxy" containerID="cri-o://1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" gracePeriod=30 Oct 30 21:46:07.951322 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:07.951252 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="method-gate" containerID="cri-o://fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" gracePeriod=30 Oct 30 21:46:08.004564 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.004531 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd"] Oct 30 21:46:08.004845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.004807 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="prom-label-proxy" containerID="cri-o://3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" gracePeriod=30 Oct 30 21:46:08.004909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.004881 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="kube-rbac-proxy" containerID="cri-o://e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" gracePeriod=30 Oct 30 21:46:08.004962 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.004807 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="method-gate" containerID="cri-o://5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" gracePeriod=30 Oct 30 21:46:08.261819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.261789 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:46:08.283498 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.283472 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:46:08.438289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438247 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config\") pod \"d6878e97-4390-4316-bd13-0675fc01b206\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " Oct 30 21:46:08.438508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438320 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config\" (UniqueName: \"kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config\") pod \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " Oct 30 21:46:08.438508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438479 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn\" (UniqueName: \"kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn\") pod \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " Oct 30 21:46:08.438648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438529 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca\") pod \"d6878e97-4390-4316-bd13-0675fc01b206\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " Oct 30 21:46:08.438648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438567 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j\" (UniqueName: \"kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j\") pod \"d6878e97-4390-4316-bd13-0675fc01b206\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " Oct 30 21:46:08.438648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438640 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca\") pod \"d6878e97-4390-4316-bd13-0675fc01b206\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " Oct 30 21:46:08.438807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438688 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls\" (UniqueName: \"kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls\") pod \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\" (UID: \"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a\") " Oct 30 21:46:08.438807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438701 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config" pod "c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" (UID: "c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:08.438807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438739 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls\" (UniqueName: \"kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls\") pod \"d6878e97-4390-4316-bd13-0675fc01b206\" (UID: \"d6878e97-4390-4316-bd13-0675fc01b206\") " Oct 30 21:46:08.438935 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.438896 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca" pod "d6878e97-4390-4316-bd13-0675fc01b206" (UID: "d6878e97-4390-4316-bd13-0675fc01b206"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:08.439096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.439032 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.439096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.439059 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.439536 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.439333 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca" pod "d6878e97-4390-4316-bd13-0675fc01b206" (UID: "d6878e97-4390-4316-bd13-0675fc01b206"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:08.439536 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.439351 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config" pod "d6878e97-4390-4316-bd13-0675fc01b206" (UID: "d6878e97-4390-4316-bd13-0675fc01b206"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:08.441001 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.440975 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls" pod "c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" (UID: "c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:08.441391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.441290 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn" pod "c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" (UID: "c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a"). InnerVolumeSpecName "kube-api-access-sdzxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:08.441391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.441369 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j" pod "d6878e97-4390-4316-bd13-0675fc01b206" (UID: "d6878e97-4390-4316-bd13-0675fc01b206"). InnerVolumeSpecName "kube-api-access-z5f8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:08.441559 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.441516 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls" pod "d6878e97-4390-4316-bd13-0675fc01b206" (UID: "d6878e97-4390-4316-bd13-0675fc01b206"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:08.539934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.539830 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzxn\" (UniqueName: \"kubernetes.io/projected/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-kube-api-access-sdzxn\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.539934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.539862 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5f8j\" (UniqueName: \"kubernetes.io/projected/d6878e97-4390-4316-bd13-0675fc01b206-kube-api-access-z5f8j\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.539934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.539872 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.539934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.539882 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.539934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.539891 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d6878e97-4390-4316-bd13-0675fc01b206-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.539934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.539900 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6878e97-4390-4316-bd13-0675fc01b206-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.649326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649291 4988 generic.go:356] "Generic (PLEG): container finished" podID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerID="3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" exitCode=0 Oct 30 21:46:08.649326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649322 4988 generic.go:356] "Generic (PLEG): container finished" podID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerID="5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" exitCode=0 Oct 30 21:46:08.649326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649331 4988 generic.go:356] "Generic (PLEG): container finished" podID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerID="e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" exitCode=0 Oct 30 21:46:08.649638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649342 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" Oct 30 21:46:08.649638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649378 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerDied","Data":"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8"} Oct 30 21:46:08.649638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649471 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerDied","Data":"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24"} Oct 30 21:46:08.649638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649513 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerDied","Data":"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f"} Oct 30 21:46:08.649638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649539 4988 scope.go:122] "RemoveContainer" containerID="3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" Oct 30 21:46:08.649638 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.649551 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd" event={"ID":"c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a","Type":"ContainerDied","Data":"17a9391c6c33cb08a6d1431c6a9c890e8aac4b7e8cebb75a9c61b276f9cc4162"} Oct 30 21:46:08.651452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651432 4988 generic.go:356] "Generic (PLEG): container finished" podID="d6878e97-4390-4316-bd13-0675fc01b206" containerID="1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" exitCode=0 Oct 30 21:46:08.651452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651451 4988 generic.go:356] "Generic (PLEG): container finished" podID="d6878e97-4390-4316-bd13-0675fc01b206" containerID="fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" exitCode=0 Oct 30 21:46:08.651637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651458 4988 generic.go:356] "Generic (PLEG): container finished" podID="d6878e97-4390-4316-bd13-0675fc01b206" containerID="5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" exitCode=0 Oct 30 21:46:08.651637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651531 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" Oct 30 21:46:08.651637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651526 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerDied","Data":"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569"} Oct 30 21:46:08.651637 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651633 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerDied","Data":"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c"} Oct 30 21:46:08.651777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651664 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerDied","Data":"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852"} Oct 30 21:46:08.651777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.651688 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4" event={"ID":"d6878e97-4390-4316-bd13-0675fc01b206","Type":"ContainerDied","Data":"e973027a5dda90c45ee349a3c8083d876b800c2c31f5e7aa2fe61305d2cbe1ce"} Oct 30 21:46:08.658849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.658828 4988 scope.go:122] "RemoveContainer" containerID="5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" Oct 30 21:46:08.667489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.667463 4988 scope.go:122] "RemoveContainer" containerID="e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" Oct 30 21:46:08.675474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.675452 4988 scope.go:122] "RemoveContainer" containerID="3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" Oct 30 21:46:08.675799 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:08.675778 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": container with ID starting with 3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8 not found: ID does not exist" containerID="3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" Oct 30 21:46:08.675863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.675811 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8"} err="failed to get container status \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": rpc error: code = NotFound desc = could not find container \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": container with ID starting with 3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8 not found: ID does not exist" Oct 30 21:46:08.675863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.675830 4988 scope.go:122] "RemoveContainer" containerID="5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" Oct 30 21:46:08.676035 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:08.676016 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": container with ID starting with 5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24 not found: ID does not exist" containerID="5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" Oct 30 21:46:08.676075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676043 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24"} err="failed to get container status \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": rpc error: code = NotFound desc = could not find container \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": container with ID starting with 5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24 not found: ID does not exist" Oct 30 21:46:08.676075 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676063 4988 scope.go:122] "RemoveContainer" containerID="e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" Oct 30 21:46:08.676310 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:08.676291 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": container with ID starting with e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f not found: ID does not exist" containerID="e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" Oct 30 21:46:08.676356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676314 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f"} err="failed to get container status \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": rpc error: code = NotFound desc = could not find container \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": container with ID starting with e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f not found: ID does not exist" Oct 30 21:46:08.676356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676326 4988 scope.go:122] "RemoveContainer" containerID="3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" Oct 30 21:46:08.676578 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676562 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8"} err="failed to get container status \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": rpc error: code = NotFound desc = could not find container \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": container with ID starting with 3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8 not found: ID does not exist" Oct 30 21:46:08.676627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676578 4988 scope.go:122] "RemoveContainer" containerID="5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" Oct 30 21:46:08.676783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676766 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24"} err="failed to get container status \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": rpc error: code = NotFound desc = could not find container \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": container with ID starting with 5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24 not found: ID does not exist" Oct 30 21:46:08.676879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676783 4988 scope.go:122] "RemoveContainer" containerID="e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" Oct 30 21:46:08.676989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676971 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f"} err="failed to get container status \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": rpc error: code = NotFound desc = could not find container \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": container with ID starting with e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f not found: ID does not exist" Oct 30 21:46:08.677036 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.676989 4988 scope.go:122] "RemoveContainer" containerID="3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8" Oct 30 21:46:08.677202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.677186 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8"} err="failed to get container status \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": rpc error: code = NotFound desc = could not find container \"3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8\": container with ID starting with 3d695e32ce9d7955bd0918bf71c841df9e30afef4986e4f66bb3b437f75c4be8 not found: ID does not exist" Oct 30 21:46:08.677253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.677202 4988 scope.go:122] "RemoveContainer" containerID="5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24" Oct 30 21:46:08.677393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.677378 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24"} err="failed to get container status \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": rpc error: code = NotFound desc = could not find container \"5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24\": container with ID starting with 5c163ab8a77fae2e1ad3ab84d011accc7639d917056aae506a43028440bbbc24 not found: ID does not exist" Oct 30 21:46:08.677458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.677393 4988 scope.go:122] "RemoveContainer" containerID="e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f" Oct 30 21:46:08.677625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.677605 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f"} err="failed to get container status \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": rpc error: code = NotFound desc = could not find container \"e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f\": container with ID starting with e5361de718e87c2a10af7e504928c87a6567d538e5d19f3831d45f7666ff927f not found: ID does not exist" Oct 30 21:46:08.677675 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.677625 4988 scope.go:122] "RemoveContainer" containerID="1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" Oct 30 21:46:08.685814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.685783 4988 scope.go:122] "RemoveContainer" containerID="fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" Oct 30 21:46:08.693563 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.693538 4988 scope.go:122] "RemoveContainer" containerID="5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" Oct 30 21:46:08.701617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.701594 4988 scope.go:122] "RemoveContainer" containerID="1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" Oct 30 21:46:08.701952 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:08.701925 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": container with ID starting with 1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569 not found: ID does not exist" containerID="1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" Oct 30 21:46:08.702009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.701992 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569"} err="failed to get container status \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": rpc error: code = NotFound desc = could not find container \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": container with ID starting with 1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569 not found: ID does not exist" Oct 30 21:46:08.702050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.702011 4988 scope.go:122] "RemoveContainer" containerID="fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" Oct 30 21:46:08.702311 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:08.702286 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": container with ID starting with fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c not found: ID does not exist" containerID="fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" Oct 30 21:46:08.702403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.702318 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c"} err="failed to get container status \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": rpc error: code = NotFound desc = could not find container \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": container with ID starting with fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c not found: ID does not exist" Oct 30 21:46:08.702403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.702334 4988 scope.go:122] "RemoveContainer" containerID="5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" Oct 30 21:46:08.702852 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:08.702834 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": container with ID starting with 5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852 not found: ID does not exist" containerID="5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" Oct 30 21:46:08.702922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.702855 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852"} err="failed to get container status \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": rpc error: code = NotFound desc = could not find container \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": container with ID starting with 5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852 not found: ID does not exist" Oct 30 21:46:08.702922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.702871 4988 scope.go:122] "RemoveContainer" containerID="1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" Oct 30 21:46:08.703121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703101 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569"} err="failed to get container status \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": rpc error: code = NotFound desc = could not find container \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": container with ID starting with 1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569 not found: ID does not exist" Oct 30 21:46:08.703166 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703125 4988 scope.go:122] "RemoveContainer" containerID="fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" Oct 30 21:46:08.703353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703327 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c"} err="failed to get container status \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": rpc error: code = NotFound desc = could not find container \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": container with ID starting with fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c not found: ID does not exist" Oct 30 21:46:08.703435 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703352 4988 scope.go:122] "RemoveContainer" containerID="5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" Oct 30 21:46:08.703600 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703578 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852"} err="failed to get container status \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": rpc error: code = NotFound desc = could not find container \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": container with ID starting with 5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852 not found: ID does not exist" Oct 30 21:46:08.703649 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703601 4988 scope.go:122] "RemoveContainer" containerID="1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569" Oct 30 21:46:08.703818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703794 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569"} err="failed to get container status \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": rpc error: code = NotFound desc = could not find container \"1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569\": container with ID starting with 1add1935b96f8107f8cbdf5f7dbaf64ee96ba41abb2a8f8c7617c98518358569 not found: ID does not exist" Oct 30 21:46:08.703862 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.703819 4988 scope.go:122] "RemoveContainer" containerID="fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c" Oct 30 21:46:08.704022 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.704002 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c"} err="failed to get container status \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": rpc error: code = NotFound desc = could not find container \"fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c\": container with ID starting with fe502e617ab500c73a1903abb74d5b683d2f2c5a1de2532ea15e1db8947d9a0c not found: ID does not exist" Oct 30 21:46:08.704063 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.704022 4988 scope.go:122] "RemoveContainer" containerID="5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852" Oct 30 21:46:08.704203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.704188 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852"} err="failed to get container status \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": rpc error: code = NotFound desc = could not find container \"5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852\": container with ID starting with 5f8023ab1984c283ba753f068b9a9ff473d2825600d431ea30b636771b599852 not found: ID does not exist" Oct 30 21:46:08.815059 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.814975 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd"] Oct 30 21:46:08.820111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.820079 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-rq5xd"] Oct 30 21:46:08.937301 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.937263 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4"] Oct 30 21:46:08.946528 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:08.946490 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-j24x4"] Oct 30 21:46:09.049463 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.049398 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:09.215648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.215616 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:09.351736 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.351702 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:46:09.352044 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.352013 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="alertmanager" containerID="cri-o://9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430" gracePeriod=120 Oct 30 21:46:09.352103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.352046 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="config-reloader" containerID="cri-o://1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e" gracePeriod=120 Oct 30 21:46:09.362317 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.362279 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:46:09.362654 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.362612 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="prometheus" containerID="cri-o://1a5dcd1d3abdc5d5265b37089ac20b74fdd2dd192c56c9cdfb9b7bdd5fce9a54" gracePeriod=600 Oct 30 21:46:09.362755 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.362687 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="thanos-sidecar" containerID="cri-o://ec6a1131c6e99546cd29e3a5f2daf0909a3639d2c22b4739d85b555ee93fb280" gracePeriod=600 Oct 30 21:46:09.362818 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.362691 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="config-reloader" containerID="cri-o://997d5ebbf294c5092ef514f2d9e66948a849d293fc439831b736fd1e3ddff5c8" gracePeriod=600 Oct 30 21:46:09.388695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.388655 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:09.554079 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.554042 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:09.662742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.662711 4988 generic.go:356] "Generic (PLEG): container finished" podID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerID="ec6a1131c6e99546cd29e3a5f2daf0909a3639d2c22b4739d85b555ee93fb280" exitCode=0 Oct 30 21:46:09.662742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.662738 4988 generic.go:356] "Generic (PLEG): container finished" podID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerID="997d5ebbf294c5092ef514f2d9e66948a849d293fc439831b736fd1e3ddff5c8" exitCode=0 Oct 30 21:46:09.662742 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.662746 4988 generic.go:356] "Generic (PLEG): container finished" podID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerID="1a5dcd1d3abdc5d5265b37089ac20b74fdd2dd192c56c9cdfb9b7bdd5fce9a54" exitCode=0 Oct 30 21:46:09.663069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.662790 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerDied","Data":"ec6a1131c6e99546cd29e3a5f2daf0909a3639d2c22b4739d85b555ee93fb280"} Oct 30 21:46:09.663069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.662852 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerDied","Data":"997d5ebbf294c5092ef514f2d9e66948a849d293fc439831b736fd1e3ddff5c8"} Oct 30 21:46:09.663069 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.662889 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerDied","Data":"1a5dcd1d3abdc5d5265b37089ac20b74fdd2dd192c56c9cdfb9b7bdd5fce9a54"} Oct 30 21:46:09.696125 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.696094 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:09.851468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851366 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.851468 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851435 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.851715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851479 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.851715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851545 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.851715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851572 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.851927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851893 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:09.852107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.851986 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:09.852107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.852031 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.852292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.852159 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.852292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.852201 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.852292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.852255 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.852508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.852376 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.852967 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.852923 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:09.853704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.853683 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.853855 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.853840 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.853958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.853946 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config\") pod \"fbbabd88-a421-4c8f-adfb-3c541f515568\" (UID: \"fbbabd88-a421-4c8f-adfb-3c541f515568\") " Oct 30 21:46:09.854335 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.854313 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.854523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.854506 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.854632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.854615 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fbbabd88-a421-4c8f-adfb-3c541f515568-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.854726 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.854700 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:09.854797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.854728 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:09.854797 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.854763 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:09.855059 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.855036 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:09.856391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.856361 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:09.857042 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.857005 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "kube-api-access-zcxfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:09.857170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.857148 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:09.857650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.857592 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:09.860239 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.860213 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:46:09.869796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.869753 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config" pod "fbbabd88-a421-4c8f-adfb-3c541f515568" (UID: "fbbabd88-a421-4c8f-adfb-3c541f515568"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:09.955849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955803 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.955849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955842 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxfx\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-kube-api-access-zcxfx\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.955849 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955853 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955864 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbbabd88-a421-4c8f-adfb-3c541f515568-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955875 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955886 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955895 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbbabd88-a421-4c8f-adfb-3c541f515568-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955905 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955914 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fbbabd88-a421-4c8f-adfb-3c541f515568-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:09.956107 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.955944 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:46:09.968584 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:09.968559 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:46:10.016501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.016469 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" path="/var/lib/kubelet/pods/c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a/volumes" Oct 30 21:46:10.016959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.016946 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6878e97-4390-4316-bd13-0675fc01b206" path="/var/lib/kubelet/pods/d6878e97-4390-4316-bd13-0675fc01b206/volumes" Oct 30 21:46:10.056938 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.056897 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:46:10.669695 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.669658 4988 generic.go:356] "Generic (PLEG): container finished" podID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerID="1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e" exitCode=0 Oct 30 21:46:10.669879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.669731 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerDied","Data":"1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e"} Oct 30 21:46:10.671975 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.671951 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"fbbabd88-a421-4c8f-adfb-3c541f515568","Type":"ContainerDied","Data":"26633a9f57c4e5c0cff4326a9f4311b44702c3e2b28b5909f65c5184c2b017c7"} Oct 30 21:46:10.672121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.671998 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:10.672121 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.672028 4988 scope.go:122] "RemoveContainer" containerID="ec6a1131c6e99546cd29e3a5f2daf0909a3639d2c22b4739d85b555ee93fb280" Oct 30 21:46:10.680774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.680747 4988 scope.go:122] "RemoveContainer" containerID="997d5ebbf294c5092ef514f2d9e66948a849d293fc439831b736fd1e3ddff5c8" Oct 30 21:46:10.689202 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.689175 4988 scope.go:122] "RemoveContainer" containerID="1a5dcd1d3abdc5d5265b37089ac20b74fdd2dd192c56c9cdfb9b7bdd5fce9a54" Oct 30 21:46:10.698099 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.698073 4988 scope.go:122] "RemoveContainer" containerID="ced10f250228ba1f402b4327ef877767ce8daab918e83b4b9abb891cc40a2ec3" Oct 30 21:46:10.840269 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.840228 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:46:10.846545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:10.846514 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:46:11.619017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.618982 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:46:11.619504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.619279 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="761906b9-f754-4d00-8a8d-4df9d5c80777" containerName="otc-container" containerID="cri-o://0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001" gracePeriod=30 Oct 30 21:46:11.688583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.688547 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.843571 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.843538 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:11.862885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.862856 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:11.872702 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.872624 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal\" (UniqueName: \"kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal\") pod \"761906b9-f754-4d00-8a8d-4df9d5c80777\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " Oct 30 21:46:11.872702 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.872663 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw\" (UniqueName: \"kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw\") pod \"761906b9-f754-4d00-8a8d-4df9d5c80777\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " Oct 30 21:46:11.872934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.872704 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs\") pod \"761906b9-f754-4d00-8a8d-4df9d5c80777\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " Oct 30 21:46:11.872934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.872814 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs\") pod \"761906b9-f754-4d00-8a8d-4df9d5c80777\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " Oct 30 21:46:11.872934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.872868 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert\") pod \"761906b9-f754-4d00-8a8d-4df9d5c80777\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " Oct 30 21:46:11.873173 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.873149 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal" pod "761906b9-f754-4d00-8a8d-4df9d5c80777" (UID: "761906b9-f754-4d00-8a8d-4df9d5c80777"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:11.875256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.875218 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs" pod "761906b9-f754-4d00-8a8d-4df9d5c80777" (UID: "761906b9-f754-4d00-8a8d-4df9d5c80777"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:11.875256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.875240 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert" pod "761906b9-f754-4d00-8a8d-4df9d5c80777" (UID: "761906b9-f754-4d00-8a8d-4df9d5c80777"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:11.875494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.875405 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs" pod "761906b9-f754-4d00-8a8d-4df9d5c80777" (UID: "761906b9-f754-4d00-8a8d-4df9d5c80777"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:11.875494 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.875431 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw" pod "761906b9-f754-4d00-8a8d-4df9d5c80777" (UID: "761906b9-f754-4d00-8a8d-4df9d5c80777"). InnerVolumeSpecName "kube-api-access-j8fsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:11.973362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.973330 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs\") pod \"761906b9-f754-4d00-8a8d-4df9d5c80777\" (UID: \"761906b9-f754-4d00-8a8d-4df9d5c80777\") " Oct 30 21:46:11.973531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.973518 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8fsw\" (UniqueName: \"kubernetes.io/projected/761906b9-f754-4d00-8a8d-4df9d5c80777-kube-api-access-j8fsw\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.973531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.973534 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.973617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.973545 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.973617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.973557 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.973617 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.973567 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/761906b9-f754-4d00-8a8d-4df9d5c80777-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:11.975851 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:11.975820 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs" pod "761906b9-f754-4d00-8a8d-4df9d5c80777" (UID: "761906b9-f754-4d00-8a8d-4df9d5c80777"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:12.018444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.017477 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" path="/var/lib/kubelet/pods/fbbabd88-a421-4c8f-adfb-3c541f515568/volumes" Oct 30 21:46:12.074493 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.074455 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/761906b9-f754-4d00-8a8d-4df9d5c80777-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:12.087484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.087434 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:12.425571 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.425439 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:46:12.461439 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.433756 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:46:12.685814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.685723 4988 generic.go:356] "Generic (PLEG): container finished" podID="761906b9-f754-4d00-8a8d-4df9d5c80777" containerID="0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001" exitCode=0 Oct 30 21:46:12.685814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.685770 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:12.685814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.685802 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"761906b9-f754-4d00-8a8d-4df9d5c80777","Type":"ContainerDied","Data":"0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001"} Oct 30 21:46:12.686365 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.685857 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"761906b9-f754-4d00-8a8d-4df9d5c80777","Type":"ContainerDied","Data":"f13d7958b83a84496f8796a4ec52c263bca50cdc45e2d8b1f586c4e8deca654c"} Oct 30 21:46:12.686365 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.685881 4988 scope.go:122] "RemoveContainer" containerID="0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001" Oct 30 21:46:12.697017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.696994 4988 scope.go:122] "RemoveContainer" containerID="0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001" Oct 30 21:46:12.697379 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:12.697356 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001\": container with ID starting with 0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001 not found: ID does not exist" containerID="0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001" Oct 30 21:46:12.697453 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.697390 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001"} err="failed to get container status \"0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001\": rpc error: code = NotFound desc = could not find container \"0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001\": container with ID starting with 0b919cb4be3d7b42f201bfaa383dbfbc0c34dd6ceee7299cff7d8f2140306001 not found: ID does not exist" Oct 30 21:46:12.850913 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.850875 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:46:12.855334 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:12.855303 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:46:13.270597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:13.270546 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.148:9093/-/ready\": dial tcp 10.132.0.148:9093: connect: connection refused" Oct 30 21:46:13.327909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:13.327872 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.475403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:13.475368 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.625729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:13.625642 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:13.772250 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:13.772214 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:14.016071 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:14.016026 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761906b9-f754-4d00-8a8d-4df9d5c80777" path="/var/lib/kubelet/pods/761906b9-f754-4d00-8a8d-4df9d5c80777/volumes" Oct 30 21:46:18.271117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:18.271070 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.148:9093/-/ready\": dial tcp 10.132.0.148:9093: connect: connection refused" Oct 30 21:46:19.592873 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.592840 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:19.717589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.717559 4988 generic.go:356] "Generic (PLEG): container finished" podID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerID="9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430" exitCode=0 Oct 30 21:46:19.717811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.717615 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:19.717811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.717648 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerDied","Data":"9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430"} Oct 30 21:46:19.717811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.717721 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0","Type":"ContainerDied","Data":"a135d857d1dedb1181af555ed678db8ced09f32ddd58e26dddcbf883aecb5a68"} Oct 30 21:46:19.717811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.717759 4988 scope.go:122] "RemoveContainer" containerID="1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e" Oct 30 21:46:19.726012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.725928 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726012 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.725973 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726014 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726068 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726072 4988 scope.go:122] "RemoveContainer" containerID="9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430" Oct 30 21:46:19.726150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726095 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726150 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726121 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726449 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726174 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn\") pod \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\" (UID: \"45ab3b5f-62ac-49c5-a70a-d201b57d9fe0\") " Oct 30 21:46:19.726666 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.726624 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:19.728763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.728734 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "kube-api-access-lkgpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:19.728763 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.728743 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:19.729327 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.729304 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:19.729507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.729248 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:19.732927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.732906 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:19.741224 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.741197 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config" pod "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" (UID: "45ab3b5f-62ac-49c5-a70a-d201b57d9fe0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:19.749708 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.749683 4988 scope.go:122] "RemoveContainer" containerID="3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40" Oct 30 21:46:19.758257 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.758231 4988 scope.go:122] "RemoveContainer" containerID="1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e" Oct 30 21:46:19.758580 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:19.758557 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e\": container with ID starting with 1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e not found: ID does not exist" containerID="1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e" Oct 30 21:46:19.758632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.758594 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e"} err="failed to get container status \"1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e\": rpc error: code = NotFound desc = could not find container \"1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e\": container with ID starting with 1bd18584d4528c22ad559e2456bdb6cfce03c0a761a6de8d063c174bde90e77e not found: ID does not exist" Oct 30 21:46:19.758632 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.758615 4988 scope.go:122] "RemoveContainer" containerID="9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430" Oct 30 21:46:19.758906 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:19.758890 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430\": container with ID starting with 9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430 not found: ID does not exist" containerID="9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430" Oct 30 21:46:19.758958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.758912 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430"} err="failed to get container status \"9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430\": rpc error: code = NotFound desc = could not find container \"9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430\": container with ID starting with 9e0ce3e92c26d917ea52ce1ff83edecc1c41af1484126ffcfd496dd0c9647430 not found: ID does not exist" Oct 30 21:46:19.758958 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.758925 4988 scope.go:122] "RemoveContainer" containerID="3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40" Oct 30 21:46:19.759132 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:19.759116 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40\": container with ID starting with 3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40 not found: ID does not exist" containerID="3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40" Oct 30 21:46:19.759176 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.759137 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40"} err="failed to get container status \"3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40\": rpc error: code = NotFound desc = could not find container \"3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40\": container with ID starting with 3591df744ca959702eb9a11eb9a57c8a82bf47b60f57ee79beb9685cf7c0ee40 not found: ID does not exist" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828115 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828166 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828182 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828200 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828214 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828237 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:19.833903 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:19.828253 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgpn\" (UniqueName: \"kubernetes.io/projected/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0-kube-api-access-lkgpn\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:20.138309 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:20.138271 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:46:20.187826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:20.187777 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:46:22.016246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:22.016215 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" path="/var/lib/kubelet/pods/45ab3b5f-62ac-49c5-a70a-d201b57d9fe0/volumes" Oct 30 21:46:25.449783 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:25.449740 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:26.394631 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:26.394585 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:46:27.121152 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:27.121113 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.313476 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:27.313440 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.495989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:27.495953 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.646527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:27.646487 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.792153 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:27.792054 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:27.942824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:27.942782 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.097260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.097175 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.140989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.140946 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn"] Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141288 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="prom-label-proxy" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141300 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="prom-label-proxy" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141312 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="method-gate" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141317 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="method-gate" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141327 4988 cpu_manager.go:414] "containerMap: removing container" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="config-reloader" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141333 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="config-reloader" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141344 4988 cpu_manager.go:414] "containerMap: removing container" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="config-reloader" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141352 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="config-reloader" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141361 4988 cpu_manager.go:414] "containerMap: removing container" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="thanos-sidecar" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141367 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="thanos-sidecar" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141375 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="kube-rbac-proxy" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141380 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="kube-rbac-proxy" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141386 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="kube-rbac-proxy" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141391 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="kube-rbac-proxy" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141403 4988 cpu_manager.go:414] "containerMap: removing container" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="prometheus" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141425 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="prometheus" Oct 30 21:46:28.141397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141432 4988 cpu_manager.go:414] "containerMap: removing container" podUID="761906b9-f754-4d00-8a8d-4df9d5c80777" containerName="otc-container" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141437 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761906b9-f754-4d00-8a8d-4df9d5c80777" containerName="otc-container" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141443 4988 cpu_manager.go:414] "containerMap: removing container" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="alertmanager" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141448 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="alertmanager" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141458 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="method-gate" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141463 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="method-gate" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141469 4988 cpu_manager.go:414] "containerMap: removing container" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="init-config-reloader" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141473 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="init-config-reloader" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141480 4988 cpu_manager.go:414] "containerMap: removing container" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="prom-label-proxy" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141486 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="prom-label-proxy" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141503 4988 cpu_manager.go:414] "containerMap: removing container" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="init-config-reloader" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141508 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="init-config-reloader" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141563 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="method-gate" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141574 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="prom-label-proxy" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141584 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="prom-label-proxy" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141590 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="method-gate" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141596 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="config-reloader" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141603 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="config-reloader" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141608 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="thanos-sidecar" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141613 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6878e97-4390-4316-bd13-0675fc01b206" containerName="kube-rbac-proxy" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141619 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cffdbb-da2b-435f-83c1-d4fa16f3ea1a" containerName="kube-rbac-proxy" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141625 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbabd88-a421-4c8f-adfb-3c541f515568" containerName="prometheus" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141630 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="761906b9-f754-4d00-8a8d-4df9d5c80777" containerName="otc-container" Oct 30 21:46:28.141968 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.141635 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ab3b5f-62ac-49c5-a70a-d201b57d9fe0" containerName="alertmanager" Oct 30 21:46:28.146379 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.146355 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.194386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.194343 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.194605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.194445 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.194605 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.194545 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.194685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.194599 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvf9j\" (UniqueName: \"kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.194726 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.194678 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.199021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.198997 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:46:28.199021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.198995 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:46:28.199219 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.199156 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:46:28.248922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.248864 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-blltv\"" Oct 30 21:46:28.248922 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.248895 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:46:28.250175 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.250148 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn"] Oct 30 21:46:28.256931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.256898 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx"] Oct 30 21:46:28.260593 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.260572 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.295242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295205 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.295242 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295245 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.295553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295327 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.295553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295362 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.295553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295389 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.295553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295457 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgqv\" (UniqueName: \"kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.295553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295495 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvf9j\" (UniqueName: \"kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.295776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.295578 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.296106 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.296084 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.297739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.297716 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.299240 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.299226 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:46:28.299303 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.299264 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:46:28.306869 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.306843 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.306986 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.306873 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.314375 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.314334 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-8cqdm\"" Oct 30 21:46:28.325062 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.325021 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx"] Oct 30 21:46:28.359441 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.359331 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvf9j\" (UniqueName: \"kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.395861 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.395820 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.396048 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.396023 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgqv\" (UniqueName: \"kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.396116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.396059 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.396116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.396077 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.399197 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.399176 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:46:28.406845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.406814 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.414964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.414939 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:46:28.419300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.419280 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.457671 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.457623 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgqv\" (UniqueName: \"kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv\") pod \"data-science-thanos-querier-proxy-77657866bd-jhjhx\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.464094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.464063 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:46:28.575051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.575007 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:28.577807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.577783 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:46:28.840884 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.840661 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn"] Oct 30 21:46:28.843812 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:46:28.843781 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed2da6d_c319_4534_8426_6da6e8fa4f2e.slice/crio-983ac173d60b6d71851dd2104f1753d1fa00879cd848455d07a19fd25f2f7dda WatchSource:0}: Error finding container 983ac173d60b6d71851dd2104f1753d1fa00879cd848455d07a19fd25f2f7dda: Status 404 returned error can't find the container with id 983ac173d60b6d71851dd2104f1753d1fa00879cd848455d07a19fd25f2f7dda Oct 30 21:46:28.956364 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:28.956318 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx"] Oct 30 21:46:28.959737 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:46:28.959703 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50026f5e_ef45_4dfe_a758_4d929119681f.slice/crio-fefd564173f54a656c1af0be9cce88b352a269bc2421b4bc4a3bfc6bdaf906f4 WatchSource:0}: Error finding container fefd564173f54a656c1af0be9cce88b352a269bc2421b4bc4a3bfc6bdaf906f4: Status 404 returned error can't find the container with id fefd564173f54a656c1af0be9cce88b352a269bc2421b4bc4a3bfc6bdaf906f4 Oct 30 21:46:29.767471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.767433 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerStarted","Data":"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d"} Oct 30 21:46:29.768015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.767512 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerStarted","Data":"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c"} Oct 30 21:46:29.768015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.767548 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerStarted","Data":"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e"} Oct 30 21:46:29.768015 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.767582 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerStarted","Data":"fefd564173f54a656c1af0be9cce88b352a269bc2421b4bc4a3bfc6bdaf906f4"} Oct 30 21:46:29.769465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.769436 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerStarted","Data":"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5"} Oct 30 21:46:29.769580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.769507 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerStarted","Data":"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e"} Oct 30 21:46:29.769580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.769534 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerStarted","Data":"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388"} Oct 30 21:46:29.769580 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.769560 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerStarted","Data":"983ac173d60b6d71851dd2104f1753d1fa00879cd848455d07a19fd25f2f7dda"} Oct 30 21:46:29.778167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.778127 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:29.889405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:29.889343 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" podStartSLOduration=1.889326479 podStartE2EDuration="1.889326479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:29.887539056 +0000 UTC m=+4217.995034967" watchObservedRunningTime="2026-10-30 21:46:29.889326479 +0000 UTC m=+4217.996822392" Oct 30 21:46:30.007432 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:30.007247 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" podStartSLOduration=2.007229385 podStartE2EDuration="2.007229385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:30.006946389 +0000 UTC m=+4218.114442338" watchObservedRunningTime="2026-10-30 21:46:30.007229385 +0000 UTC m=+4218.114725294" Oct 30 21:46:32.914088 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:32.914049 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:32.942685 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:32.942646 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:46:32.946230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:32.946209 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.009359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.009315 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:46:33.035835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.035798 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.036021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.035866 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.036021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.035958 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcjc\" (UniqueName: \"kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.036021 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.036010 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.036188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.036112 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.036188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.036148 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.052993 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.052957 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:46:33.053176 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.053121 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dczck\"" Oct 30 21:46:33.053217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.053195 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:46:33.053267 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.053249 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:46:33.056878 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.056853 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:46:33.057013 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.056903 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:46:33.070683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.070647 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.137279 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.137230 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.137485 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.137386 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.137562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.137479 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.137562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.137514 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcjc\" (UniqueName: \"kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.137562 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.137540 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.137704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.137614 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.138745 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.138696 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.140400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.140373 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.140400 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.140395 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.141181 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.141155 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.141247 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.141190 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.200093 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.200016 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcjc\" (UniqueName: \"kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc\") pod \"data-science-collector-collector-1\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.222215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.222179 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.259186 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.259148 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:46:33.379543 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.379504 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.642674 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.642640 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:46:33.646220 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:46:33.646191 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cd8802_6a0a_45c9_a7a9_79eb92a01375.slice/crio-a7f7acc2c294f0c285b63f347951e1e433634d122ff19e02cf9aec007a250a6d WatchSource:0}: Error finding container a7f7acc2c294f0c285b63f347951e1e433634d122ff19e02cf9aec007a250a6d: Status 404 returned error can't find the container with id a7f7acc2c294f0c285b63f347951e1e433634d122ff19e02cf9aec007a250a6d Oct 30 21:46:33.648487 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.648454 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:33.648569 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.648519 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:33.790187 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.790149 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74cd8802-6a0a-45c9-a7a9-79eb92a01375","Type":"ContainerStarted","Data":"24a002afe155a5d948783b53fec02543ecf98eae1bd410d2bab9a4155d9d0389"} Oct 30 21:46:33.790451 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.790200 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74cd8802-6a0a-45c9-a7a9-79eb92a01375","Type":"ContainerStarted","Data":"a7f7acc2c294f0c285b63f347951e1e433634d122ff19e02cf9aec007a250a6d"} Oct 30 21:46:33.911009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:33.910902 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.910888631 podStartE2EDuration="1.910888631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:33.910293056 +0000 UTC m=+4222.017788971" watchObservedRunningTime="2026-10-30 21:46:33.910888631 +0000 UTC m=+4222.018384543" Oct 30 21:46:45.584601 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:45.584557 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:45.926835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:45.926799 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.115386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:46.115344 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.264838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:46.264744 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.865116 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:46.865078 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.035497 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.035462 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:46:47.039423 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.039390 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.076019 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.075973 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:47.144043 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.144000 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:46:47.144244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.144001 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:46:47.144244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.144001 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:46:47.144244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.144011 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:46:47.144244 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.144026 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-6xrfq\"" Oct 30 21:46:47.145491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.145470 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:46:47.149486 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.149460 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:46:47.149625 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.149570 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.161087 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161054 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.161230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161103 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.161230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161190 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.161230 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161225 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.161466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161250 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.161466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161269 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.161466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.161368 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6v9k\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.196474 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.196436 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:46:47.234947 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.234911 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:46:47.235126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.235027 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:46:47.235217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.235199 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:46:47.235217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.235212 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:46:47.235392 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.235380 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:46:47.235477 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.235463 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:46:47.235536 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.235525 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-vfzqg\"" Oct 30 21:46:47.262417 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262383 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.262517 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262474 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.262517 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262500 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.262597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262525 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.262690 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262664 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.262758 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262742 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6v9k\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.262815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.262790 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.263083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.263049 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.264989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.264964 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.265201 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.265183 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.265270 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.265222 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.265525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.265504 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.265525 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.265520 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.285795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.285737 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:46:47.285795 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.285746 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:46:47.286057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.285821 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:46:47.324274 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.324237 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6v9k\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.357157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.357119 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:47.363644 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363609 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.363744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363657 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.363744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363714 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.363744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363733 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.363885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363762 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.363885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363846 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.363964 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.363895 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.364027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.364008 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.364064 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.364043 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.364117 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.364073 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.364177 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.364159 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkclg\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.364255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.364239 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465353 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465392 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465450 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465479 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465498 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465531 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465551 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465584 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465644 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkclg\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.465707 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465704 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.466086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465730 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.466086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.465759 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.466611 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.466586 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.467642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.467589 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.467793 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.467764 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.468788 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.468741 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.468894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.468873 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.468974 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.468897 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.469039 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.469009 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.469108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.469055 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.469548 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.469525 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.469548 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.469540 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.469731 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.469712 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.528639 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.528604 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkclg\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:47.788346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.788311 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:46:47.790941 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:46:47.790910 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3db7550_ab30_4176_8fe6_1311a006bb15.slice/crio-b2660dd02936bd2c693829b5604ebbd9a9992c4351aa005042de8916e21e3283 WatchSource:0}: Error finding container b2660dd02936bd2c693829b5604ebbd9a9992c4351aa005042de8916e21e3283: Status 404 returned error can't find the container with id b2660dd02936bd2c693829b5604ebbd9a9992c4351aa005042de8916e21e3283 Oct 30 21:46:47.853911 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:47.853875 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerStarted","Data":"b2660dd02936bd2c693829b5604ebbd9a9992c4351aa005042de8916e21e3283"} Oct 30 21:46:48.860879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:48.860783 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerStarted","Data":"37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d"} Oct 30 21:46:51.604888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.604841 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:51.709832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.709795 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:51.807542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.807503 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:51.807754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.807633 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:51.860484 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.860380 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:51.876552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.876520 4988 generic.go:356] "Generic (PLEG): container finished" podID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerID="37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d" exitCode=0 Oct 30 21:46:51.876748 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:51.876593 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerDied","Data":"37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d"} Oct 30 21:46:52.014776 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.014742 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:52.080680 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.080647 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:52.270240 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.270200 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:46:52.756696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.756657 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:46:52.760384 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:46:52.760342 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3f8761_d01f_4dd3_a1ca_f7c9b1a80bbe.slice/crio-258bcb25b1454c118b27a5e219be7f0f0818c64a2b6fa71b28db844325c9ee9d WatchSource:0}: Error finding container 258bcb25b1454c118b27a5e219be7f0f0818c64a2b6fa71b28db844325c9ee9d: Status 404 returned error can't find the container with id 258bcb25b1454c118b27a5e219be7f0f0818c64a2b6fa71b28db844325c9ee9d Oct 30 21:46:52.883542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.883497 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerStarted","Data":"cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d"} Oct 30 21:46:52.883716 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.883599 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerStarted","Data":"a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04"} Oct 30 21:46:52.883840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.883813 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:52.884786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:52.884763 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerStarted","Data":"258bcb25b1454c118b27a5e219be7f0f0818c64a2b6fa71b28db844325c9ee9d"} Oct 30 21:46:53.008158 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:53.008099 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.622519101 podStartE2EDuration="7.008082718s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385563617s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:53.006585384 +0000 UTC m=+4241.114081314" watchObservedRunningTime="2026-10-30 21:46:53.008082718 +0000 UTC m=+4241.115578659" Oct 30 21:46:53.891343 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:53.891306 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerStarted","Data":"d6cb2b36808b38d50b6a962dcce091e5091062c2876606c74270eee18d48dbe1"} Oct 30 21:46:56.395328 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:46:56.395284 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:46:57.910895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:57.910855 4988 generic.go:356] "Generic (PLEG): container finished" podID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerID="d6cb2b36808b38d50b6a962dcce091e5091062c2876606c74270eee18d48dbe1" exitCode=0 Oct 30 21:46:57.911308 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:57.910929 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerDied","Data":"d6cb2b36808b38d50b6a962dcce091e5091062c2876606c74270eee18d48dbe1"} Oct 30 21:46:58.892396 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:58.892365 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:46:58.918619 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:58.918586 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerStarted","Data":"923d6cd11334d729c328863e43bd597a0933a9cdce4ed8c289dd2eee2fde5f17"} Oct 30 21:46:58.919068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:58.918653 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerStarted","Data":"763057b923ed6379b0b4a6983c0a9f747b6b03742b633fcc6a67f3241f5e31a5"} Oct 30 21:46:58.919068 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:58.918702 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerStarted","Data":"08af0d717ac3e43c0b7a9f5cde8d9fdacaad8ecb6d5edbca780fe1fac4d12433"} Oct 30 21:46:59.150223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:46:59.150120 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=8.365212742 podStartE2EDuration="12.150104399s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784891657s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:59.148058897 +0000 UTC m=+4247.255554800" watchObservedRunningTime="2026-10-30 21:46:59.150104399 +0000 UTC m=+4247.257600312" Oct 30 21:47:00.610829 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:00.610782 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:02.271160 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:02.271118 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:47:07.271023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:07.270977 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:47:07.276893 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:07.276865 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:47:07.966118 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:07.966089 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:47:13.908902 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:13.908863 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.053733 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:14.053696 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.259111 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:14.259015 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.454489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:14.454448 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.599223 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:14.599141 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:14.823346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:14.823309 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:19.681222 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:19.681184 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:19.681665 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:19.681380 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="74cd8802-6a0a-45c9-a7a9-79eb92a01375" containerName="otc-container" containerID="cri-o://24a002afe155a5d948783b53fec02543ecf98eae1bd410d2bab9a4155d9d0389" gracePeriod=30 Oct 30 21:47:19.988814 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:19.988719 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.134354 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:20.134315 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:20.358233 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:20.358142 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:22.025749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.025713 4988 generic.go:356] "Generic (PLEG): container finished" podID="74cd8802-6a0a-45c9-a7a9-79eb92a01375" containerID="24a002afe155a5d948783b53fec02543ecf98eae1bd410d2bab9a4155d9d0389" exitCode=0 Oct 30 21:47:22.026123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.025784 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74cd8802-6a0a-45c9-a7a9-79eb92a01375","Type":"ContainerDied","Data":"24a002afe155a5d948783b53fec02543ecf98eae1bd410d2bab9a4155d9d0389"} Oct 30 21:47:22.026123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.025851 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"74cd8802-6a0a-45c9-a7a9-79eb92a01375","Type":"ContainerDied","Data":"a7f7acc2c294f0c285b63f347951e1e433634d122ff19e02cf9aec007a250a6d"} Oct 30 21:47:22.026123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.025881 4988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7f7acc2c294f0c285b63f347951e1e433634d122ff19e02cf9aec007a250a6d" Oct 30 21:47:22.032590 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.032565 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:22.178811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.178765 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs\") pod \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " Oct 30 21:47:22.179032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.178843 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs\") pod \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " Oct 30 21:47:22.179032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.178918 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal\" (UniqueName: \"kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal\") pod \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " Oct 30 21:47:22.179032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.178982 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert\") pod \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " Oct 30 21:47:22.179032 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.179016 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc\" (UniqueName: \"kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc\") pod \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " Oct 30 21:47:22.179285 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.179064 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs\") pod \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\" (UID: \"74cd8802-6a0a-45c9-a7a9-79eb92a01375\") " Oct 30 21:47:22.179351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.179314 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal" pod "74cd8802-6a0a-45c9-a7a9-79eb92a01375" (UID: "74cd8802-6a0a-45c9-a7a9-79eb92a01375"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:22.179739 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.179471 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:22.181405 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.181376 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs" pod "74cd8802-6a0a-45c9-a7a9-79eb92a01375" (UID: "74cd8802-6a0a-45c9-a7a9-79eb92a01375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:22.181561 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.181538 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs" pod "74cd8802-6a0a-45c9-a7a9-79eb92a01375" (UID: "74cd8802-6a0a-45c9-a7a9-79eb92a01375"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:22.181633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.181613 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert" pod "74cd8802-6a0a-45c9-a7a9-79eb92a01375" (UID: "74cd8802-6a0a-45c9-a7a9-79eb92a01375"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:22.181840 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.181817 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs" pod "74cd8802-6a0a-45c9-a7a9-79eb92a01375" (UID: "74cd8802-6a0a-45c9-a7a9-79eb92a01375"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:22.181979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.181963 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc" pod "74cd8802-6a0a-45c9-a7a9-79eb92a01375" (UID: "74cd8802-6a0a-45c9-a7a9-79eb92a01375"). InnerVolumeSpecName "kube-api-access-tmcjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:22.280007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.279968 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:22.280007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.280000 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcjc\" (UniqueName: \"kubernetes.io/projected/74cd8802-6a0a-45c9-a7a9-79eb92a01375-kube-api-access-tmcjc\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:22.280007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.280012 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:22.280258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.280022 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:22.280258 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:22.280032 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd8802-6a0a-45c9-a7a9-79eb92a01375-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:23.030190 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.030154 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.194128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.194091 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:23.243312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.243273 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:23.278539 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.278502 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:23.278889 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.278876 4988 cpu_manager.go:414] "containerMap: removing container" podUID="74cd8802-6a0a-45c9-a7a9-79eb92a01375" containerName="otc-container" Oct 30 21:47:23.278929 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.278890 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74cd8802-6a0a-45c9-a7a9-79eb92a01375" containerName="otc-container" Oct 30 21:47:23.278969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.278961 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74cd8802-6a0a-45c9-a7a9-79eb92a01375" containerName="otc-container" Oct 30 21:47:23.282181 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.282137 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.334864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.334834 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:47:23.334864 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.334837 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:47:23.335102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.334902 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:47:23.335102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.334916 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:47:23.335102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.334922 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:47:23.335102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.334945 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dczck\"" Oct 30 21:47:23.341919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.341887 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:23.392199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.392154 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.392199 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.392209 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.392450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.392297 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.392450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.392355 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.392450 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.392396 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.392561 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.392510 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwn2r\" (UniqueName: \"kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.493592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.493538 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.493592 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.493605 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.493882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.493638 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.493882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.493652 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.493882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.493676 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwn2r\" (UniqueName: \"kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.493882 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.493770 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.494504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.494478 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.496294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.496272 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.496386 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.496323 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.496552 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.496536 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.496692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.496671 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.555701 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.555614 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwn2r\" (UniqueName: \"kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r\") pod \"data-science-collector-collector-1\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.595175 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.595134 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:23.984722 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.984686 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:23.988928 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:47:23.988888 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd360ef86_d6aa_41f0_8dea_b25d5fcfb274.slice/crio-776c187a7339891fda8666212759735c3878face7aa0c43a3fe008309da914e3 WatchSource:0}: Error finding container 776c187a7339891fda8666212759735c3878face7aa0c43a3fe008309da914e3: Status 404 returned error can't find the container with id 776c187a7339891fda8666212759735c3878face7aa0c43a3fe008309da914e3 Oct 30 21:47:23.991055 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.991018 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:23.991163 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:23.991084 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:24.015523 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:24.015492 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74cd8802-6a0a-45c9-a7a9-79eb92a01375" path="/var/lib/kubelet/pods/74cd8802-6a0a-45c9-a7a9-79eb92a01375/volumes" Oct 30 21:47:24.035664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:24.035624 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d360ef86-d6aa-41f0-8dea-b25d5fcfb274","Type":"ContainerStarted","Data":"776c187a7339891fda8666212759735c3878face7aa0c43a3fe008309da914e3"} Oct 30 21:47:25.042103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:25.042065 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d360ef86-d6aa-41f0-8dea-b25d5fcfb274","Type":"ContainerStarted","Data":"35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df"} Oct 30 21:47:25.843025 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:25.842986 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:26.394182 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:47:26.394148 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:47:45.667721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:45.667679 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:45.865141 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:45.865107 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:46.013294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:46.013205 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:46.214234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:46.214188 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:46.362275 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:46.362180 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:46.520459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:46.520394 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:51.513648 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:51.513587 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=28.513572679 podStartE2EDuration="28.513572679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:25.199350966 +0000 UTC m=+4273.306846879" watchObservedRunningTime="2026-10-30 21:47:51.513572679 +0000 UTC m=+4299.621068590" Oct 30 21:47:51.515108 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:51.515080 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:51.515371 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:51.515347 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d360ef86-d6aa-41f0-8dea-b25d5fcfb274" containerName="otc-container" containerID="cri-o://35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df" gracePeriod=30 Oct 30 21:47:51.865504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:51.865402 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:52.064752 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:52.064710 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:52.222941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:52.222900 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:53.875574 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.875535 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:53.970370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970266 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs\") pod \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " Oct 30 21:47:53.970370 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970328 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs\") pod \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " Oct 30 21:47:53.970662 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970436 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r\" (UniqueName: \"kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r\") pod \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " Oct 30 21:47:53.970662 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970511 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert\") pod \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " Oct 30 21:47:53.970662 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970538 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal\") pod \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " Oct 30 21:47:53.970662 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970580 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs\") pod \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\" (UID: \"d360ef86-d6aa-41f0-8dea-b25d5fcfb274\") " Oct 30 21:47:53.971009 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.970972 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal" pod "d360ef86-d6aa-41f0-8dea-b25d5fcfb274" (UID: "d360ef86-d6aa-41f0-8dea-b25d5fcfb274"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:53.972762 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.972715 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert" pod "d360ef86-d6aa-41f0-8dea-b25d5fcfb274" (UID: "d360ef86-d6aa-41f0-8dea-b25d5fcfb274"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:53.972927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.972767 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs" pod "d360ef86-d6aa-41f0-8dea-b25d5fcfb274" (UID: "d360ef86-d6aa-41f0-8dea-b25d5fcfb274"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:53.972927 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.972826 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r" pod "d360ef86-d6aa-41f0-8dea-b25d5fcfb274" (UID: "d360ef86-d6aa-41f0-8dea-b25d5fcfb274"). InnerVolumeSpecName "kube-api-access-qwn2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:53.973046 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.973004 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs" pod "d360ef86-d6aa-41f0-8dea-b25d5fcfb274" (UID: "d360ef86-d6aa-41f0-8dea-b25d5fcfb274"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:53.973103 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:53.973048 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs" pod "d360ef86-d6aa-41f0-8dea-b25d5fcfb274" (UID: "d360ef86-d6aa-41f0-8dea-b25d5fcfb274"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:54.071703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.071646 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.071703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.071701 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.072004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.071714 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.072004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.071726 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.072004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.071735 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.072004 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.071746 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwn2r\" (UniqueName: \"kubernetes.io/projected/d360ef86-d6aa-41f0-8dea-b25d5fcfb274-kube-api-access-qwn2r\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.188192 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.188149 4988 generic.go:356] "Generic (PLEG): container finished" podID="d360ef86-d6aa-41f0-8dea-b25d5fcfb274" containerID="35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df" exitCode=0 Oct 30 21:47:54.188376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.188198 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.188376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.188239 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d360ef86-d6aa-41f0-8dea-b25d5fcfb274","Type":"ContainerDied","Data":"35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df"} Oct 30 21:47:54.188376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.188301 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d360ef86-d6aa-41f0-8dea-b25d5fcfb274","Type":"ContainerDied","Data":"776c187a7339891fda8666212759735c3878face7aa0c43a3fe008309da914e3"} Oct 30 21:47:54.188376 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.188330 4988 scope.go:122] "RemoveContainer" containerID="35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df" Oct 30 21:47:54.197248 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.197226 4988 scope.go:122] "RemoveContainer" containerID="35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df" Oct 30 21:47:54.197598 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:47:54.197571 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df\": container with ID starting with 35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df not found: ID does not exist" containerID="35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df" Oct 30 21:47:54.197700 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.197606 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df"} err="failed to get container status \"35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df\": rpc error: code = NotFound desc = could not find container \"35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df\": container with ID starting with 35919e5c33fe0da0c2e9f8a0c81569fad1135d983004e597a8711d579c5a82df not found: ID does not exist" Oct 30 21:47:54.356622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.356533 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:54.363796 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.363769 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:54.457969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.457935 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:54.458346 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.458331 4988 cpu_manager.go:414] "containerMap: removing container" podUID="d360ef86-d6aa-41f0-8dea-b25d5fcfb274" containerName="otc-container" Oct 30 21:47:54.458393 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.458347 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d360ef86-d6aa-41f0-8dea-b25d5fcfb274" containerName="otc-container" Oct 30 21:47:54.458453 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.458432 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="d360ef86-d6aa-41f0-8dea-b25d5fcfb274" containerName="otc-container" Oct 30 21:47:54.462790 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.462766 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.524066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.524027 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 21:47:54.524066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.524027 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 21:47:54.524066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.524041 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dczck\"" Oct 30 21:47:54.524066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.524048 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 21:47:54.524399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.524078 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 21:47:54.524399 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.524108 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 21:47:54.538203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.538164 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:54.575621 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.575573 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.575836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.575634 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7nfq\" (UniqueName: \"kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.575836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.575794 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.575836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.575828 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.575990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.575850 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.575990 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.575880 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.677356 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.677316 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.677573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.677365 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7nfq\" (UniqueName: \"kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.677573 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.677550 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.677656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.677594 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.677729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.677707 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.677787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.677767 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.678385 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.678358 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.680137 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.680110 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.680243 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.680194 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.680342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.680321 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.680381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.680327 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.743100 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.743064 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7nfq\" (UniqueName: \"kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq\") pod \"data-science-collector-collector-1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:54.776614 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:54.776570 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:47:55.163667 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:55.163621 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:47:55.166404 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:47:55.166375 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcece0fd8_2fae_40ab_a8bf_a4112cbaf9f1.slice/crio-996c53f58f6dd2168bf478cac01909d7c47ed6c5a888738d234f62cd7b5ca3b3 WatchSource:0}: Error finding container 996c53f58f6dd2168bf478cac01909d7c47ed6c5a888738d234f62cd7b5ca3b3: Status 404 returned error can't find the container with id 996c53f58f6dd2168bf478cac01909d7c47ed6c5a888738d234f62cd7b5ca3b3 Oct 30 21:47:55.171177 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:55.171131 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:55.171351 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:55.171199 4988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:55.198909 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:55.198867 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1","Type":"ContainerStarted","Data":"996c53f58f6dd2168bf478cac01909d7c47ed6c5a888738d234f62cd7b5ca3b3"} Oct 30 21:47:56.015729 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:56.015689 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d360ef86-d6aa-41f0-8dea-b25d5fcfb274" path="/var/lib/kubelet/pods/d360ef86-d6aa-41f0-8dea-b25d5fcfb274/volumes" Oct 30 21:47:56.206260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:56.206222 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1","Type":"ContainerStarted","Data":"870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f"} Oct 30 21:47:56.325183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:47:56.325078 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.325064291 podStartE2EDuration="2.325064291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:56.324077079 +0000 UTC m=+4304.431573005" watchObservedRunningTime="2026-10-30 21:47:56.325064291 +0000 UTC m=+4304.432560201" Oct 30 21:47:56.402694 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:47:56.402646 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:48:08.659095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.659049 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn"] Oct 30 21:48:08.659704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.659362 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="kube-rbac-proxy" containerID="cri-o://6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" gracePeriod=30 Oct 30 21:48:08.659704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.659458 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="prom-label-proxy" containerID="cri-o://67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" gracePeriod=30 Oct 30 21:48:08.659704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.659463 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="method-gate" containerID="cri-o://e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" gracePeriod=30 Oct 30 21:48:08.713007 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.712972 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx"] Oct 30 21:48:08.713241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.713216 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="kube-rbac-proxy" containerID="cri-o://ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" gracePeriod=30 Oct 30 21:48:08.713308 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.713229 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="prom-label-proxy" containerID="cri-o://3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" gracePeriod=30 Oct 30 21:48:08.713359 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.713225 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="method-gate" containerID="cri-o://e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" gracePeriod=30 Oct 30 21:48:08.959305 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.959273 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:48:08.978951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:08.978918 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:48:09.118281 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118243 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config\") pod \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " Oct 30 21:48:09.118507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118307 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca\") pod \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " Oct 30 21:48:09.118507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118365 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv\" (UniqueName: \"kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv\") pod \"50026f5e-ef45-4dfe-a758-4d929119681f\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " Oct 30 21:48:09.118507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118403 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls\" (UniqueName: \"kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls\") pod \"50026f5e-ef45-4dfe-a758-4d929119681f\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " Oct 30 21:48:09.118507 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118468 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j\" (UniqueName: \"kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j\") pod \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " Oct 30 21:48:09.118749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118513 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls\" (UniqueName: \"kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls\") pod \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " Oct 30 21:48:09.118749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118565 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config\" (UniqueName: \"kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config\") pod \"50026f5e-ef45-4dfe-a758-4d929119681f\" (UID: \"50026f5e-ef45-4dfe-a758-4d929119681f\") " Oct 30 21:48:09.118749 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118622 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca\") pod \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\" (UID: \"4ed2da6d-c319-4534-8426-6da6e8fa4f2e\") " Oct 30 21:48:09.118919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118777 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config" pod "4ed2da6d-c319-4534-8426-6da6e8fa4f2e" (UID: "4ed2da6d-c319-4534-8426-6da6e8fa4f2e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:09.118919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118862 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca" pod "4ed2da6d-c319-4534-8426-6da6e8fa4f2e" (UID: "4ed2da6d-c319-4534-8426-6da6e8fa4f2e"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:09.118919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.118911 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.119083 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.119038 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca" pod "4ed2da6d-c319-4534-8426-6da6e8fa4f2e" (UID: "4ed2da6d-c319-4534-8426-6da6e8fa4f2e"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:09.119143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.119123 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config" pod "50026f5e-ef45-4dfe-a758-4d929119681f" (UID: "50026f5e-ef45-4dfe-a758-4d929119681f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:09.120982 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.120945 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls" pod "4ed2da6d-c319-4534-8426-6da6e8fa4f2e" (UID: "4ed2da6d-c319-4534-8426-6da6e8fa4f2e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:09.120982 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.120960 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv" pod "50026f5e-ef45-4dfe-a758-4d929119681f" (UID: "50026f5e-ef45-4dfe-a758-4d929119681f"). InnerVolumeSpecName "kube-api-access-mtgqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:09.121343 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.121318 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls" pod "50026f5e-ef45-4dfe-a758-4d929119681f" (UID: "50026f5e-ef45-4dfe-a758-4d929119681f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:09.121472 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.121349 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j" pod "4ed2da6d-c319-4534-8426-6da6e8fa4f2e" (UID: "4ed2da6d-c319-4534-8426-6da6e8fa4f2e"). InnerVolumeSpecName "kube-api-access-lvf9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220270 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50026f5e-ef45-4dfe-a758-4d929119681f-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220298 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220308 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220319 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgqv\" (UniqueName: \"kubernetes.io/projected/50026f5e-ef45-4dfe-a758-4d929119681f-kube-api-access-mtgqv\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220329 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/50026f5e-ef45-4dfe-a758-4d929119681f-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220338 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvf9j\" (UniqueName: \"kubernetes.io/projected/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-kube-api-access-lvf9j\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.220358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.220347 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4ed2da6d-c319-4534-8426-6da6e8fa4f2e-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.269987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.269952 4988 generic.go:356] "Generic (PLEG): container finished" podID="50026f5e-ef45-4dfe-a758-4d929119681f" containerID="3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" exitCode=0 Oct 30 21:48:09.269987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.269979 4988 generic.go:356] "Generic (PLEG): container finished" podID="50026f5e-ef45-4dfe-a758-4d929119681f" containerID="e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" exitCode=0 Oct 30 21:48:09.269987 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.269986 4988 generic.go:356] "Generic (PLEG): container finished" podID="50026f5e-ef45-4dfe-a758-4d929119681f" containerID="ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" exitCode=0 Oct 30 21:48:09.270312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.270018 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" Oct 30 21:48:09.270312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.270033 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerDied","Data":"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d"} Oct 30 21:48:09.270312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.270105 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerDied","Data":"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c"} Oct 30 21:48:09.270312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.270142 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerDied","Data":"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e"} Oct 30 21:48:09.270312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.270173 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx" event={"ID":"50026f5e-ef45-4dfe-a758-4d929119681f","Type":"ContainerDied","Data":"fefd564173f54a656c1af0be9cce88b352a269bc2421b4bc4a3bfc6bdaf906f4"} Oct 30 21:48:09.270312 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.270174 4988 scope.go:122] "RemoveContainer" containerID="3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" Oct 30 21:48:09.272263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272236 4988 generic.go:356] "Generic (PLEG): container finished" podID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerID="67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" exitCode=0 Oct 30 21:48:09.272263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272258 4988 generic.go:356] "Generic (PLEG): container finished" podID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerID="e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" exitCode=0 Oct 30 21:48:09.272263 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272267 4988 generic.go:356] "Generic (PLEG): container finished" podID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerID="6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" exitCode=0 Oct 30 21:48:09.272577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272267 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerDied","Data":"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5"} Oct 30 21:48:09.272577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272303 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" Oct 30 21:48:09.272577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272333 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerDied","Data":"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e"} Oct 30 21:48:09.272577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272360 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerDied","Data":"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388"} Oct 30 21:48:09.272577 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.272388 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn" event={"ID":"4ed2da6d-c319-4534-8426-6da6e8fa4f2e","Type":"ContainerDied","Data":"983ac173d60b6d71851dd2104f1753d1fa00879cd848455d07a19fd25f2f7dda"} Oct 30 21:48:09.280813 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.280793 4988 scope.go:122] "RemoveContainer" containerID="e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" Oct 30 21:48:09.290195 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.290171 4988 scope.go:122] "RemoveContainer" containerID="ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" Oct 30 21:48:09.298535 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.298509 4988 scope.go:122] "RemoveContainer" containerID="3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" Oct 30 21:48:09.298856 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:09.298833 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": container with ID starting with 3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d not found: ID does not exist" containerID="3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" Oct 30 21:48:09.298969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.298865 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d"} err="failed to get container status \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": rpc error: code = NotFound desc = could not find container \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": container with ID starting with 3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d not found: ID does not exist" Oct 30 21:48:09.298969 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.298885 4988 scope.go:122] "RemoveContainer" containerID="e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" Oct 30 21:48:09.299102 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:09.299088 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": container with ID starting with e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c not found: ID does not exist" containerID="e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" Oct 30 21:48:09.299149 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299106 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c"} err="failed to get container status \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": rpc error: code = NotFound desc = could not find container \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": container with ID starting with e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c not found: ID does not exist" Oct 30 21:48:09.299149 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299118 4988 scope.go:122] "RemoveContainer" containerID="ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" Oct 30 21:48:09.299313 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:09.299297 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": container with ID starting with ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e not found: ID does not exist" containerID="ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" Oct 30 21:48:09.299349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299318 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e"} err="failed to get container status \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": rpc error: code = NotFound desc = could not find container \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": container with ID starting with ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e not found: ID does not exist" Oct 30 21:48:09.299349 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299330 4988 scope.go:122] "RemoveContainer" containerID="3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" Oct 30 21:48:09.299593 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299574 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d"} err="failed to get container status \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": rpc error: code = NotFound desc = could not find container \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": container with ID starting with 3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d not found: ID does not exist" Oct 30 21:48:09.299691 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299592 4988 scope.go:122] "RemoveContainer" containerID="e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" Oct 30 21:48:09.299809 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299791 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c"} err="failed to get container status \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": rpc error: code = NotFound desc = could not find container \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": container with ID starting with e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c not found: ID does not exist" Oct 30 21:48:09.299852 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299810 4988 scope.go:122] "RemoveContainer" containerID="ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" Oct 30 21:48:09.299989 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299974 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e"} err="failed to get container status \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": rpc error: code = NotFound desc = could not find container \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": container with ID starting with ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e not found: ID does not exist" Oct 30 21:48:09.300034 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.299989 4988 scope.go:122] "RemoveContainer" containerID="3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d" Oct 30 21:48:09.300207 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.300191 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d"} err="failed to get container status \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": rpc error: code = NotFound desc = could not find container \"3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d\": container with ID starting with 3a6825d1b6be3c87b4f0f3af44776c42aff3b78a356bc333a21a449b3fa1039d not found: ID does not exist" Oct 30 21:48:09.300254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.300207 4988 scope.go:122] "RemoveContainer" containerID="e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c" Oct 30 21:48:09.300485 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.300392 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c"} err="failed to get container status \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": rpc error: code = NotFound desc = could not find container \"e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c\": container with ID starting with e1119abdbd0f41d41fae9afeec7c3272013a0b238465aeaca5e2aa02eeb2304c not found: ID does not exist" Oct 30 21:48:09.300485 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.300430 4988 scope.go:122] "RemoveContainer" containerID="ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e" Oct 30 21:48:09.300642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.300625 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e"} err="failed to get container status \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": rpc error: code = NotFound desc = could not find container \"ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e\": container with ID starting with ca22281bec0d0670844515ad9682b38207af0e2e1e79937c328e758d324baf9e not found: ID does not exist" Oct 30 21:48:09.300692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.300643 4988 scope.go:122] "RemoveContainer" containerID="67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" Oct 30 21:48:09.309241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.309220 4988 scope.go:122] "RemoveContainer" containerID="e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" Oct 30 21:48:09.317430 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.317393 4988 scope.go:122] "RemoveContainer" containerID="6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" Oct 30 21:48:09.325807 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.325780 4988 scope.go:122] "RemoveContainer" containerID="67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" Oct 30 21:48:09.326134 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:09.326111 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": container with ID starting with 67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5 not found: ID does not exist" containerID="67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" Oct 30 21:48:09.326217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.326141 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5"} err="failed to get container status \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": rpc error: code = NotFound desc = could not find container \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": container with ID starting with 67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5 not found: ID does not exist" Oct 30 21:48:09.326217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.326163 4988 scope.go:122] "RemoveContainer" containerID="e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" Oct 30 21:48:09.326472 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:09.326452 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": container with ID starting with e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e not found: ID does not exist" containerID="e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" Oct 30 21:48:09.326553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.326484 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e"} err="failed to get container status \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": rpc error: code = NotFound desc = could not find container \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": container with ID starting with e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e not found: ID does not exist" Oct 30 21:48:09.326553 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.326509 4988 scope.go:122] "RemoveContainer" containerID="6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" Oct 30 21:48:09.326782 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:09.326765 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": container with ID starting with 6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388 not found: ID does not exist" containerID="6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" Oct 30 21:48:09.326825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.326788 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388"} err="failed to get container status \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": rpc error: code = NotFound desc = could not find container \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": container with ID starting with 6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388 not found: ID does not exist" Oct 30 21:48:09.326825 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.326803 4988 scope.go:122] "RemoveContainer" containerID="67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" Oct 30 21:48:09.327023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327006 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5"} err="failed to get container status \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": rpc error: code = NotFound desc = could not find container \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": container with ID starting with 67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5 not found: ID does not exist" Oct 30 21:48:09.327071 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327023 4988 scope.go:122] "RemoveContainer" containerID="e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" Oct 30 21:48:09.327253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327235 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e"} err="failed to get container status \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": rpc error: code = NotFound desc = could not find container \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": container with ID starting with e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e not found: ID does not exist" Oct 30 21:48:09.327326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327254 4988 scope.go:122] "RemoveContainer" containerID="6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" Oct 30 21:48:09.327555 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327539 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388"} err="failed to get container status \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": rpc error: code = NotFound desc = could not find container \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": container with ID starting with 6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388 not found: ID does not exist" Oct 30 21:48:09.327616 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327555 4988 scope.go:122] "RemoveContainer" containerID="67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5" Oct 30 21:48:09.327838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327818 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5"} err="failed to get container status \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": rpc error: code = NotFound desc = could not find container \"67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5\": container with ID starting with 67502c7c038f7ac2669d5532990643222930b124fa7b3245a8fff1bb49f739d5 not found: ID does not exist" Oct 30 21:48:09.327838 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.327836 4988 scope.go:122] "RemoveContainer" containerID="e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e" Oct 30 21:48:09.328043 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.328028 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e"} err="failed to get container status \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": rpc error: code = NotFound desc = could not find container \"e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e\": container with ID starting with e34ecece866bc311edaed46aa021534ca0342e4f6ec586838da448a71bca929e not found: ID does not exist" Oct 30 21:48:09.328093 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.328043 4988 scope.go:122] "RemoveContainer" containerID="6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388" Oct 30 21:48:09.328232 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.328216 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388"} err="failed to get container status \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": rpc error: code = NotFound desc = could not find container \"6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388\": container with ID starting with 6f1de8b89a5b8420dc468c29edca5e3e2f082c3a557c3e26017aabbe7ffaf388 not found: ID does not exist" Oct 30 21:48:09.435096 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.435055 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx"] Oct 30 21:48:09.441112 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.441080 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-jhjhx"] Oct 30 21:48:09.551091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.551007 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn"] Oct 30 21:48:09.555253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.555211 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-cx5cn"] Oct 30 21:48:09.757465 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.757430 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:09.957704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:09.957673 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.017490 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.017455 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" path="/var/lib/kubelet/pods/4ed2da6d-c319-4534-8426-6da6e8fa4f2e/volumes" Oct 30 21:48:10.018169 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.018145 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" path="/var/lib/kubelet/pods/50026f5e-ef45-4dfe-a758-4d929119681f/volumes" Oct 30 21:48:10.040290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.040255 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:48:10.040642 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.040612 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="prometheus" containerID="cri-o://08af0d717ac3e43c0b7a9f5cde8d9fdacaad8ecb6d5edbca780fe1fac4d12433" gracePeriod=600 Oct 30 21:48:10.040774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.040637 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="thanos-sidecar" containerID="cri-o://923d6cd11334d729c328863e43bd597a0933a9cdce4ed8c289dd2eee2fde5f17" gracePeriod=600 Oct 30 21:48:10.040774 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.040658 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="config-reloader" containerID="cri-o://763057b923ed6379b0b4a6983c0a9f747b6b03742b633fcc6a67f3241f5e31a5" gracePeriod=600 Oct 30 21:48:10.048123 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.048083 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:48:10.048397 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.048364 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="alertmanager" containerID="cri-o://a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04" gracePeriod=120 Oct 30 21:48:10.048511 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.048478 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="config-reloader" containerID="cri-o://cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d" gracePeriod=120 Oct 30 21:48:10.121950 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.121912 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.265894 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.265861 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.292149 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.292110 4988 generic.go:356] "Generic (PLEG): container finished" podID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerID="cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d" exitCode=0 Oct 30 21:48:10.292329 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.292163 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerDied","Data":"cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d"} Oct 30 21:48:10.295167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.295098 4988 generic.go:356] "Generic (PLEG): container finished" podID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerID="923d6cd11334d729c328863e43bd597a0933a9cdce4ed8c289dd2eee2fde5f17" exitCode=0 Oct 30 21:48:10.295167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.295117 4988 generic.go:356] "Generic (PLEG): container finished" podID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerID="763057b923ed6379b0b4a6983c0a9f747b6b03742b633fcc6a67f3241f5e31a5" exitCode=0 Oct 30 21:48:10.295167 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.295124 4988 generic.go:356] "Generic (PLEG): container finished" podID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerID="08af0d717ac3e43c0b7a9f5cde8d9fdacaad8ecb6d5edbca780fe1fac4d12433" exitCode=0 Oct 30 21:48:10.295391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.295183 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerDied","Data":"923d6cd11334d729c328863e43bd597a0933a9cdce4ed8c289dd2eee2fde5f17"} Oct 30 21:48:10.295391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.295231 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerDied","Data":"763057b923ed6379b0b4a6983c0a9f747b6b03742b633fcc6a67f3241f5e31a5"} Oct 30 21:48:10.295391 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.295254 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerDied","Data":"08af0d717ac3e43c0b7a9f5cde8d9fdacaad8ecb6d5edbca780fe1fac4d12433"} Oct 30 21:48:10.312210 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.312184 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:10.432545 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432511 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432662 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432697 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432730 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432751 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432776 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432806 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432828 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432858 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.432930 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432884 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.433255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432934 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.433255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432931 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:10.433255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.432974 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.433255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.433027 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\" (UID: \"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe\") " Oct 30 21:48:10.433255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.433242 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:10.433652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.433271 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.433789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.433764 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:10.435756 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.435724 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "kube-api-access-qkclg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:10.436300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.436067 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.436300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.436085 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:10.436300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.436157 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.436300 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.436187 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.436583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.436369 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.436583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.436454 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:10.437459 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.437389 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.438608 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.438589 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:48:10.451380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.451340 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config" pod "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" (UID: "9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534064 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534098 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534109 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534118 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534129 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534138 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534149 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkclg\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-kube-api-access-qkclg\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534157 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534166 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534175 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.534560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534205 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:48:10.534560 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.534215 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:10.545885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.545860 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:48:10.634660 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:10.634613 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:48:11.302091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.302050 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe","Type":"ContainerDied","Data":"258bcb25b1454c118b27a5e219be7f0f0818c64a2b6fa71b28db844325c9ee9d"} Oct 30 21:48:11.302091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.302080 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:11.302548 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.302125 4988 scope.go:122] "RemoveContainer" containerID="923d6cd11334d729c328863e43bd597a0933a9cdce4ed8c289dd2eee2fde5f17" Oct 30 21:48:11.310925 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.310900 4988 scope.go:122] "RemoveContainer" containerID="763057b923ed6379b0b4a6983c0a9f747b6b03742b633fcc6a67f3241f5e31a5" Oct 30 21:48:11.319819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.319797 4988 scope.go:122] "RemoveContainer" containerID="08af0d717ac3e43c0b7a9f5cde8d9fdacaad8ecb6d5edbca780fe1fac4d12433" Oct 30 21:48:11.328734 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.328710 4988 scope.go:122] "RemoveContainer" containerID="d6cb2b36808b38d50b6a962dcce091e5091062c2876606c74270eee18d48dbe1" Oct 30 21:48:11.520095 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.520052 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:48:11.530292 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:11.530260 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:48:12.016010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.015970 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" path="/var/lib/kubelet/pods/9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe/volumes" Oct 30 21:48:12.362910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.355843 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:48:12.363381 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.363191 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" containerName="otc-container" containerID="cri-o://870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f" gracePeriod=30 Oct 30 21:48:12.403215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.403169 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.551475 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.551371 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.622531 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.622461 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:48:12.752388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.752012 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert\") pod \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " Oct 30 21:48:12.752388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.752242 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal\") pod \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " Oct 30 21:48:12.752388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.752266 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs\") pod \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " Oct 30 21:48:12.752388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.752328 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs\") pod \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " Oct 30 21:48:12.752388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.752362 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs\") pod \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " Oct 30 21:48:12.752863 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.752400 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq\" (UniqueName: \"kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq\") pod \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\" (UID: \"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1\") " Oct 30 21:48:12.753589 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.753538 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal" pod "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" (UID: "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:12.756017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.755694 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs" pod "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" (UID: "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:12.756017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.755831 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs" pod "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" (UID: "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:12.756017 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.755893 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq" pod "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" (UID: "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1"). InnerVolumeSpecName "kube-api-access-x7nfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:12.756283 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.756050 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs" pod "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" (UID: "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:12.757035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.757009 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert" pod "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" (UID: "cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:12.781491 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.781346 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:12.853284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.853240 4988 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-data-science-collector-ta-client-cert\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:12.853284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.853276 4988 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otc-internal\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:12.853284 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.853290 4988 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-otlp-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:12.853576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.853300 4988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:12.853576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.853309 4988 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-monitoring-tls-certs\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:12.853576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:12.853318 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7nfq\" (UniqueName: \"kubernetes.io/projected/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1-kube-api-access-x7nfq\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:13.314677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.314585 4988 generic.go:356] "Generic (PLEG): container finished" podID="cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" containerID="870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f" exitCode=0 Oct 30 21:48:13.314677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.314632 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 21:48:13.314888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.314665 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1","Type":"ContainerDied","Data":"870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f"} Oct 30 21:48:13.314888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.314733 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1","Type":"ContainerDied","Data":"996c53f58f6dd2168bf478cac01909d7c47ed6c5a888738d234f62cd7b5ca3b3"} Oct 30 21:48:13.314888 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.314757 4988 scope.go:122] "RemoveContainer" containerID="870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f" Oct 30 21:48:13.323837 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.323814 4988 scope.go:122] "RemoveContainer" containerID="870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f" Oct 30 21:48:13.324144 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:13.324122 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f\": container with ID starting with 870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f not found: ID does not exist" containerID="870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f" Oct 30 21:48:13.324217 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.324156 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f"} err="failed to get container status \"870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f\": rpc error: code = NotFound desc = could not find container \"870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f\": container with ID starting with 870a760d5053c06995aa61cf4cb5f4ac5d936fea071f50254330a92f2520547f not found: ID does not exist" Oct 30 21:48:13.482427 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.482387 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:48:13.532010 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.531974 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 21:48:13.892575 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:13.892538 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.154:9093/-/ready\": dial tcp 10.132.0.154:9093: connect: connection refused" Oct 30 21:48:14.016443 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:14.016391 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" path="/var/lib/kubelet/pods/cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1/volumes" Oct 30 21:48:14.067290 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:14.067239 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.281190 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:14.281088 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.433687 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:14.433648 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.649041 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:14.649005 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.769023 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:16.768985 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:18.892805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:18.892763 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.154:9093/-/ready\": dial tcp 10.132.0.154:9093: connect: connection refused" Oct 30 21:48:20.294895 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.294868 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:20.313623 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313535 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.313623 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313581 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.313623 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313612 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.313924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313768 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.313924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313801 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.313924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313830 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.313924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313879 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k\") pod \"a3db7550-ab30-4176-8fe6-1311a006bb15\" (UID: \"a3db7550-ab30-4176-8fe6-1311a006bb15\") " Oct 30 21:48:20.314139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.313948 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:20.314200 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.314167 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.315940 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.315910 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:20.316225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.316202 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:20.316347 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.316216 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:20.316489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.316469 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "kube-api-access-z6v9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:20.320291 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.320256 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:20.328140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.328108 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config" pod "a3db7550-ab30-4176-8fe6-1311a006bb15" (UID: "a3db7550-ab30-4176-8fe6-1311a006bb15"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:20.357680 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.357642 4988 generic.go:356] "Generic (PLEG): container finished" podID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerID="a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04" exitCode=0 Oct 30 21:48:20.357836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.357698 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:20.357836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.357709 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerDied","Data":"a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04"} Oct 30 21:48:20.357836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.357769 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"a3db7550-ab30-4176-8fe6-1311a006bb15","Type":"ContainerDied","Data":"b2660dd02936bd2c693829b5604ebbd9a9992c4351aa005042de8916e21e3283"} Oct 30 21:48:20.357836 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.357796 4988 scope.go:122] "RemoveContainer" containerID="cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d" Oct 30 21:48:20.369455 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.369435 4988 scope.go:122] "RemoveContainer" containerID="a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04" Oct 30 21:48:20.377583 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.377561 4988 scope.go:122] "RemoveContainer" containerID="37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d" Oct 30 21:48:20.385091 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.385064 4988 scope.go:122] "RemoveContainer" containerID="cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d" Oct 30 21:48:20.385359 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:20.385338 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d\": container with ID starting with cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d not found: ID does not exist" containerID="cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d" Oct 30 21:48:20.385447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.385369 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d"} err="failed to get container status \"cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d\": rpc error: code = NotFound desc = could not find container \"cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d\": container with ID starting with cf20d48ce83507d63f4b4436dc11c02a6c9969bc04ad5a076de8df46f421396d not found: ID does not exist" Oct 30 21:48:20.385447 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.385388 4988 scope.go:122] "RemoveContainer" containerID="a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04" Oct 30 21:48:20.385653 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:20.385638 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04\": container with ID starting with a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04 not found: ID does not exist" containerID="a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04" Oct 30 21:48:20.385692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.385658 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04"} err="failed to get container status \"a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04\": rpc error: code = NotFound desc = could not find container \"a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04\": container with ID starting with a612c41a3028924920a58c27bc5fb85e05b6bb0c16f40c9d6ad46f4b43826a04 not found: ID does not exist" Oct 30 21:48:20.385692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.385672 4988 scope.go:122] "RemoveContainer" containerID="37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d" Oct 30 21:48:20.385917 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:20.385900 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d\": container with ID starting with 37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d not found: ID does not exist" containerID="37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d" Oct 30 21:48:20.385959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.385924 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d"} err="failed to get container status \"37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d\": rpc error: code = NotFound desc = could not find container \"37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d\": container with ID starting with 37d6c65b563ced25ac99535a402e342794fa9798a8d7b2464d0139c0b6dcf16d not found: ID does not exist" Oct 30 21:48:20.414711 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.414678 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.414711 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.414708 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.414901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.414719 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.414901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.414729 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6v9k\" (UniqueName: \"kubernetes.io/projected/a3db7550-ab30-4176-8fe6-1311a006bb15-kube-api-access-z6v9k\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.414901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.414739 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3db7550-ab30-4176-8fe6-1311a006bb15-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.414901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.414747 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3db7550-ab30-4176-8fe6-1311a006bb15-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:20.632304 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.632246 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:48:20.645055 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:20.645025 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:48:22.014976 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:22.014940 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" path="/var/lib/kubelet/pods/a3db7550-ab30-4176-8fe6-1311a006bb15/volumes" Oct 30 21:48:26.129798 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:26.129759 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.395267 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:26.395233 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:48:32.697445 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:32.697384 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:32.978650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:32.978563 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.170297 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.170257 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.358947 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.358855 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.537656 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.537617 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.741262 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.741228 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.822127 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822086 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t"] Oct 30 21:48:33.822444 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822430 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="kube-rbac-proxy" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822445 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="kube-rbac-proxy" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822455 4988 cpu_manager.go:414] "containerMap: removing container" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="prom-label-proxy" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822460 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="prom-label-proxy" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822470 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="method-gate" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822476 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="method-gate" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822484 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="init-config-reloader" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822490 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="init-config-reloader" Oct 30 21:48:33.822501 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822499 4988 cpu_manager.go:414] "containerMap: removing container" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="kube-rbac-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822504 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="kube-rbac-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822510 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822515 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822520 4988 cpu_manager.go:414] "containerMap: removing container" podUID="cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" containerName="otc-container" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822524 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" containerName="otc-container" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822531 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="init-config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822535 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="init-config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822545 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822549 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822555 4988 cpu_manager.go:414] "containerMap: removing container" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="method-gate" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822559 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="method-gate" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822566 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="thanos-sidecar" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822570 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="thanos-sidecar" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822584 4988 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="prometheus" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822588 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="prometheus" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822595 4988 cpu_manager.go:414] "containerMap: removing container" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="prom-label-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822599 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="prom-label-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822605 4988 cpu_manager.go:414] "containerMap: removing container" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="alertmanager" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822609 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="alertmanager" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822659 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="kube-rbac-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822666 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822672 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="cece0fd8-2fae-40ab-a8bf-a4112cbaf9f1" containerName="otc-container" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822679 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="config-reloader" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822684 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="method-gate" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822690 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="thanos-sidecar" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822697 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f8761-d01f-4dd3-a1ca-f7c9b1a80bbe" containerName="prometheus" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822702 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="prom-label-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822707 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3db7550-ab30-4176-8fe6-1311a006bb15" containerName="alertmanager" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822713 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="kube-rbac-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822718 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="50026f5e-ef45-4dfe-a758-4d929119681f" containerName="prom-label-proxy" Oct 30 21:48:33.822787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.822725 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed2da6d-c319-4534-8426-6da6e8fa4f2e" containerName="method-gate" Oct 30 21:48:33.829035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.829006 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:33.884932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.884892 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:48:33.884932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.884893 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:48:33.919458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.919400 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:33.919672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.919511 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:33.919672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.919576 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:33.919672 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.919620 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:33.919799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.919680 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg9b\" (UniqueName: \"kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:33.933587 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.933554 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 21:48:33.934273 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.934254 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:48:33.934453 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.934440 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 21:48:33.934516 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.934493 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-fkgcs\"" Oct 30 21:48:33.934624 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.934612 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 21:48:33.935826 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.935803 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t"] Oct 30 21:48:33.945268 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.945240 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7"] Oct 30 21:48:33.948966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:33.948951 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.002380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.002291 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-85lm4\"" Oct 30 21:48:34.002380 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.002326 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 21:48:34.003027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.003006 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 21:48:34.016542 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.016511 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7"] Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020132 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020191 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020220 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020228 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nv26\" (UniqueName: \"kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020331 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020363 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg9b\" (UniqueName: \"kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020464 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020492 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.020567 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.020517 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.021215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.021130 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.021215 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.021159 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.021353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.021329 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.022959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.022936 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.086253 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.086217 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg9b\" (UniqueName: \"kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b\") pod \"data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.121056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.121013 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nv26\" (UniqueName: \"kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.121056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.121055 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.121341 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.121128 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.121943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.121920 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.123615 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.123585 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.147859 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.147822 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:48:34.182597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.182562 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nv26\" (UniqueName: \"kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26\") pod \"data-science-thanos-querier-proxy-77657866bd-2wmc7\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.268917 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.268805 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:48:34.531361 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.531285 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t"] Oct 30 21:48:34.535202 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:48:34.535171 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d5e605_e3b7_43f8_9573_d93071ac5b93.slice/crio-aa0db33f8c179da16aab3f1c00638448fec0a9c2a237c11a92c1e209e19371b0 WatchSource:0}: Error finding container aa0db33f8c179da16aab3f1c00638448fec0a9c2a237c11a92c1e209e19371b0: Status 404 returned error can't find the container with id aa0db33f8c179da16aab3f1c00638448fec0a9c2a237c11a92c1e209e19371b0 Oct 30 21:48:34.646110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:34.646081 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7"] Oct 30 21:48:35.431307 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.431271 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerStarted","Data":"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89"} Oct 30 21:48:35.431880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.431374 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerStarted","Data":"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0"} Oct 30 21:48:35.431880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.431438 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerStarted","Data":"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079"} Oct 30 21:48:35.431880 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.431467 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerStarted","Data":"5f804177d847935fcecb865e6e41eb7ef8fc10a19fd2cf05ded6b3b1debe38be"} Oct 30 21:48:35.433353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.433325 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerStarted","Data":"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913"} Oct 30 21:48:35.433504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.433387 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerStarted","Data":"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473"} Oct 30 21:48:35.433504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.433423 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerStarted","Data":"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8"} Oct 30 21:48:35.433504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.433444 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerStarted","Data":"aa0db33f8c179da16aab3f1c00638448fec0a9c2a237c11a92c1e209e19371b0"} Oct 30 21:48:35.442832 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.442802 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:35.550139 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.550078 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" podStartSLOduration=2.550064993 podStartE2EDuration="2.550064993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:35.548699316 +0000 UTC m=+4343.656195251" watchObservedRunningTime="2026-10-30 21:48:35.550064993 +0000 UTC m=+4343.657560903" Oct 30 21:48:35.668860 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:35.668788 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" podStartSLOduration=2.668772273 podStartE2EDuration="2.668772273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:35.66680748 +0000 UTC m=+4343.774303390" watchObservedRunningTime="2026-10-30 21:48:35.668772273 +0000 UTC m=+4343.776268182" Oct 30 21:48:38.565907 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:38.565870 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:38.723924 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:38.723884 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:38.877951 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:38.877865 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:39.024157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:39.024118 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.210901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:51.210863 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.541035 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:51.540944 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.705403 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:51.705367 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.865203 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:51.865075 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.473390 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.473357 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.626503 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.626466 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:52.654387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.654345 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:48:52.657998 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.657975 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.715696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.715658 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:48:52.719582 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.719549 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.765452 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.765402 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:48:52.774622 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.774591 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 21:48:52.774817 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.774778 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 21:48:52.776446 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.776428 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 21:48:52.813297 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813261 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813262 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813314 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813321 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813351 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813455 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813471 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813463 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 21:48:52.813544 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813523 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-969ts\"" Oct 30 21:48:52.813819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813549 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 21:48:52.813819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813613 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-dm52q\"" Oct 30 21:48:52.813819 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.813682 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 21:48:52.814912 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.814891 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:48:52.881604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881560 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.881604 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881610 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.881879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881636 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.881879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881734 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.881879 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881772 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.882005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881869 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881900 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqcn\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881963 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.881982 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882005 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882003 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882054 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882086 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99wq\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.882218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882131 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.882218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882149 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882169 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882218 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882187 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882237 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.882473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882285 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.882473 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.882310 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.982867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.982782 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.982867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.982822 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.982867 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.982843 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.982964 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983004 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99wq\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983037 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983067 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983098 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983128 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983162 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983193 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983220 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983246 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983286 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983315 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983347 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983395 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.983504 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983496 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.984094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983526 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqcn\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.984094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983581 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.984094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.983986 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.984094 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.984068 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.985154 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.985115 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.986383 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.986321 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.986866 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.986762 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.987618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.987111 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.987618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.987342 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.987618 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.987541 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.987860 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.987622 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.988293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.988176 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.988293 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.988206 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.988735 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.988708 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.988848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.988794 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.988981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.988965 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:52.989684 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.989661 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:52.989768 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:52.989662 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:53.044454 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:53.044398 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99wq\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:53.044610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:53.044398 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqcn\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:53.275857 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:53.275763 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:53.713844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:53.713807 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:48:53.745952 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:48:53.745920 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9be458_1a6c_4bb9_a7b5_36f18adcb228.slice/crio-a9aebdeaf43c37b4b2b0bfa17d9affb7d2e5d3a9eb7f961d380c143fcafef0cd WatchSource:0}: Error finding container a9aebdeaf43c37b4b2b0bfa17d9affb7d2e5d3a9eb7f961d380c143fcafef0cd: Status 404 returned error can't find the container with id a9aebdeaf43c37b4b2b0bfa17d9affb7d2e5d3a9eb7f961d380c143fcafef0cd Oct 30 21:48:54.526081 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:54.526041 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerStarted","Data":"415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8"} Oct 30 21:48:54.526308 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:54.526103 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerStarted","Data":"a9aebdeaf43c37b4b2b0bfa17d9affb7d2e5d3a9eb7f961d380c143fcafef0cd"} Oct 30 21:48:55.002596 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.002550 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.106704 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.106668 4988 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.204495 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.204432 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.204692 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.204522 4988 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.256342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.256260 4988 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.393714 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.393674 4988 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6dcee51846c65e61e42d030ad34b511007d41c5710f282a3474b3969662e5f6c/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.458919 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.458890 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:55.741576 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:55.741533 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:48:56.275822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:56.275791 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:48:56.280260 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:48:56.280233 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1f9485_e328_43c5_92a0_d1fd090f846c.slice/crio-074b771c965ecd63e0f60fadcbe4281333602ea009ca8a325294696bedf1a92a WatchSource:0}: Error finding container 074b771c965ecd63e0f60fadcbe4281333602ea009ca8a325294696bedf1a92a: Status 404 returned error can't find the container with id 074b771c965ecd63e0f60fadcbe4281333602ea009ca8a325294696bedf1a92a Oct 30 21:48:56.395499 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:48:56.395465 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:48:56.539427 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:56.539320 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerStarted","Data":"074b771c965ecd63e0f60fadcbe4281333602ea009ca8a325294696bedf1a92a"} Oct 30 21:48:57.545787 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:57.545745 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerStarted","Data":"edc868e8a2fe6cc31a91f6ebcb95ab49eb47927a80eeaf465ea60f532ed99eba"} Oct 30 21:48:57.547458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:57.547431 4988 generic.go:356] "Generic (PLEG): container finished" podID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerID="415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8" exitCode=0 Oct 30 21:48:57.547582 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:57.547485 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerDied","Data":"415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8"} Oct 30 21:48:58.554526 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:58.554485 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerStarted","Data":"e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6"} Oct 30 21:48:58.555043 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:58.554551 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerStarted","Data":"04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168"} Oct 30 21:48:58.555043 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:58.554894 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:48:58.681326 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:48:58.681271 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.296126351 podStartE2EDuration="6.681255056s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385128705s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:58.679752606 +0000 UTC m=+4366.787248508" watchObservedRunningTime="2026-10-30 21:48:58.681255056 +0000 UTC m=+4366.788750969" Oct 30 21:49:00.566260 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:00.566224 4988 generic.go:356] "Generic (PLEG): container finished" podID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerID="edc868e8a2fe6cc31a91f6ebcb95ab49eb47927a80eeaf465ea60f532ed99eba" exitCode=0 Oct 30 21:49:00.566683 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:00.566271 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerDied","Data":"edc868e8a2fe6cc31a91f6ebcb95ab49eb47927a80eeaf465ea60f532ed99eba"} Oct 30 21:49:01.573868 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:01.573830 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerStarted","Data":"177b906f10655232c69c1d0d60ed603e006d11c2a2f09b67d13fd766ba82d0ca"} Oct 30 21:49:01.574254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:01.573903 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerStarted","Data":"7f65bf2aaee4b02210120b7c1d218b493efac85b9fb28ad20806033248cd64df"} Oct 30 21:49:01.574254 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:01.573929 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerStarted","Data":"41ef28da10cb47aac6e9453c1b1038bde2a3e6044c8f9c43654bf90f8f69985c"} Oct 30 21:49:01.751225 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:01.751161 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.065711787 podStartE2EDuration="9.751146147s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68543436s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:01.700508939 +0000 UTC m=+4369.808004850" watchObservedRunningTime="2026-10-30 21:49:01.751146147 +0000 UTC m=+4369.858642097" Oct 30 21:49:04.561308 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:04.561274 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:49:05.742271 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:05.742232 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:49:10.742389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:10.742347 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:49:10.747997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:10.747970 4988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:49:11.628241 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:11.628210 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:49:12.947104 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:12.947069 4988 scope.go:122] "RemoveContainer" containerID="fc838f7395ffef69a1d53db743215be48d120632e77a7df3f6ed8ce612a72f54" Oct 30 21:49:26.395528 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:26.395490 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:49:26.493805 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:26.493770 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.117234 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.117183 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t"] Oct 30 21:49:33.117715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.117566 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="kube-rbac-proxy" containerID="cri-o://507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" gracePeriod=30 Oct 30 21:49:33.117715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.117620 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="prom-label-proxy" containerID="cri-o://c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" gracePeriod=30 Oct 30 21:49:33.117715 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.117673 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="method-gate" containerID="cri-o://a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" gracePeriod=30 Oct 30 21:49:33.171815 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.171781 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7"] Oct 30 21:49:33.172056 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.172031 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="prom-label-proxy" containerID="cri-o://c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" gracePeriod=30 Oct 30 21:49:33.172140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.172063 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="method-gate" containerID="cri-o://650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" gracePeriod=30 Oct 30 21:49:33.172140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.172026 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="kube-rbac-proxy" containerID="cri-o://04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" gracePeriod=30 Oct 30 21:49:33.426155 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.426121 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:49:33.443398 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.443365 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:49:33.537228 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537191 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b\" (UniqueName: \"kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b\") pod \"34d5e605-e3b7-43f8-9573-d93071ac5b93\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " Oct 30 21:49:33.537466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537255 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config\") pod \"34d5e605-e3b7-43f8-9573-d93071ac5b93\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " Oct 30 21:49:33.537466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537307 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls\" (UniqueName: \"kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls\") pod \"34d5e605-e3b7-43f8-9573-d93071ac5b93\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " Oct 30 21:49:33.537466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537333 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26\" (UniqueName: \"kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26\") pod \"06693e03-b579-4de8-b9cf-99f2cfae2a46\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " Oct 30 21:49:33.537466 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537361 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config\" (UniqueName: \"kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config\") pod \"06693e03-b579-4de8-b9cf-99f2cfae2a46\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " Oct 30 21:49:33.537718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537477 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca\") pod \"34d5e605-e3b7-43f8-9573-d93071ac5b93\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " Oct 30 21:49:33.537718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537522 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca\") pod \"34d5e605-e3b7-43f8-9573-d93071ac5b93\" (UID: \"34d5e605-e3b7-43f8-9573-d93071ac5b93\") " Oct 30 21:49:33.537718 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537571 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls\" (UniqueName: \"kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls\") pod \"06693e03-b579-4de8-b9cf-99f2cfae2a46\" (UID: \"06693e03-b579-4de8-b9cf-99f2cfae2a46\") " Oct 30 21:49:33.537885 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537786 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config" pod "34d5e605-e3b7-43f8-9573-d93071ac5b93" (UID: "34d5e605-e3b7-43f8-9573-d93071ac5b93"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:33.538003 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.537975 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca" pod "34d5e605-e3b7-43f8-9573-d93071ac5b93" (UID: "34d5e605-e3b7-43f8-9573-d93071ac5b93"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:33.538085 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.538058 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca" pod "34d5e605-e3b7-43f8-9573-d93071ac5b93" (UID: "34d5e605-e3b7-43f8-9573-d93071ac5b93"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:33.538178 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.538148 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config" pod "06693e03-b579-4de8-b9cf-99f2cfae2a46" (UID: "06693e03-b579-4de8-b9cf-99f2cfae2a46"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:33.539627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.539598 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26" pod "06693e03-b579-4de8-b9cf-99f2cfae2a46" (UID: "06693e03-b579-4de8-b9cf-99f2cfae2a46"). InnerVolumeSpecName "kube-api-access-5nv26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:33.539627 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.539619 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b" pod "34d5e605-e3b7-43f8-9573-d93071ac5b93" (UID: "34d5e605-e3b7-43f8-9573-d93071ac5b93"). InnerVolumeSpecName "kube-api-access-2wg9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:33.539769 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.539618 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls" pod "34d5e605-e3b7-43f8-9573-d93071ac5b93" (UID: "34d5e605-e3b7-43f8-9573-d93071ac5b93"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:33.539769 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.539698 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls" pod "06693e03-b579-4de8-b9cf-99f2cfae2a46" (UID: "06693e03-b579-4de8-b9cf-99f2cfae2a46"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:33.639246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639210 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639242 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/34d5e605-e3b7-43f8-9573-d93071ac5b93-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639246 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639253 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nv26\" (UniqueName: \"kubernetes.io/projected/06693e03-b579-4de8-b9cf-99f2cfae2a46-kube-api-access-5nv26\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639264 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06693e03-b579-4de8-b9cf-99f2cfae2a46-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639274 4988 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-client-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639283 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e605-e3b7-43f8-9573-d93071ac5b93-prometheus-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639292 4988 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/06693e03-b579-4de8-b9cf-99f2cfae2a46-tls\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.639532 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.639301 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wg9b\" (UniqueName: \"kubernetes.io/projected/34d5e605-e3b7-43f8-9573-d93071ac5b93-kube-api-access-2wg9b\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:33.725443 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725391 4988 generic.go:356] "Generic (PLEG): container finished" podID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerID="c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" exitCode=0 Oct 30 21:49:33.725443 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725433 4988 generic.go:356] "Generic (PLEG): container finished" podID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerID="a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" exitCode=0 Oct 30 21:49:33.725443 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725444 4988 generic.go:356] "Generic (PLEG): container finished" podID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerID="507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" exitCode=0 Oct 30 21:49:33.725789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725460 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" Oct 30 21:49:33.725789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725455 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerDied","Data":"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913"} Oct 30 21:49:33.725789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725533 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerDied","Data":"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473"} Oct 30 21:49:33.725789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725562 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerDied","Data":"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8"} Oct 30 21:49:33.725789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725597 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t" event={"ID":"34d5e605-e3b7-43f8-9573-d93071ac5b93","Type":"ContainerDied","Data":"aa0db33f8c179da16aab3f1c00638448fec0a9c2a237c11a92c1e209e19371b0"} Oct 30 21:49:33.725789 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.725636 4988 scope.go:122] "RemoveContainer" containerID="c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" Oct 30 21:49:33.728086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728061 4988 generic.go:356] "Generic (PLEG): container finished" podID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerID="c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" exitCode=0 Oct 30 21:49:33.728086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728082 4988 generic.go:356] "Generic (PLEG): container finished" podID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerID="650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" exitCode=0 Oct 30 21:49:33.728086 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728091 4988 generic.go:356] "Generic (PLEG): container finished" podID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerID="04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" exitCode=0 Oct 30 21:49:33.728289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728122 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" Oct 30 21:49:33.728289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728149 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerDied","Data":"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89"} Oct 30 21:49:33.728289 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728230 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerDied","Data":"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0"} Oct 30 21:49:33.728389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728304 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerDied","Data":"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079"} Oct 30 21:49:33.728389 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.728342 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7" event={"ID":"06693e03-b579-4de8-b9cf-99f2cfae2a46","Type":"ContainerDied","Data":"5f804177d847935fcecb865e6e41eb7ef8fc10a19fd2cf05ded6b3b1debe38be"} Oct 30 21:49:33.735652 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.735623 4988 scope.go:122] "RemoveContainer" containerID="a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" Oct 30 21:49:33.744824 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.744793 4988 scope.go:122] "RemoveContainer" containerID="507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" Oct 30 21:49:33.753382 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.753353 4988 scope.go:122] "RemoveContainer" containerID="c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" Oct 30 21:49:33.753732 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:33.753711 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": container with ID starting with c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913 not found: ID does not exist" containerID="c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" Oct 30 21:49:33.753843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.753746 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913"} err="failed to get container status \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": rpc error: code = NotFound desc = could not find container \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": container with ID starting with c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913 not found: ID does not exist" Oct 30 21:49:33.753843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.753776 4988 scope.go:122] "RemoveContainer" containerID="a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" Oct 30 21:49:33.754054 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:33.754035 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": container with ID starting with a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473 not found: ID does not exist" containerID="a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" Oct 30 21:49:33.754126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.754061 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473"} err="failed to get container status \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": rpc error: code = NotFound desc = could not find container \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": container with ID starting with a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473 not found: ID does not exist" Oct 30 21:49:33.754126 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.754081 4988 scope.go:122] "RemoveContainer" containerID="507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" Oct 30 21:49:33.754325 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:33.754302 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": container with ID starting with 507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8 not found: ID does not exist" containerID="507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" Oct 30 21:49:33.754368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.754331 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8"} err="failed to get container status \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": rpc error: code = NotFound desc = could not find container \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": container with ID starting with 507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8 not found: ID does not exist" Oct 30 21:49:33.754368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.754347 4988 scope.go:122] "RemoveContainer" containerID="c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" Oct 30 21:49:33.754931 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.754909 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913"} err="failed to get container status \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": rpc error: code = NotFound desc = could not find container \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": container with ID starting with c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913 not found: ID does not exist" Oct 30 21:49:33.754988 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.754932 4988 scope.go:122] "RemoveContainer" containerID="a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" Oct 30 21:49:33.755204 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755179 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473"} err="failed to get container status \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": rpc error: code = NotFound desc = could not find container \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": container with ID starting with a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473 not found: ID does not exist" Oct 30 21:49:33.755286 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755205 4988 scope.go:122] "RemoveContainer" containerID="507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" Oct 30 21:49:33.755467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755436 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8"} err="failed to get container status \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": rpc error: code = NotFound desc = could not find container \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": container with ID starting with 507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8 not found: ID does not exist" Oct 30 21:49:33.755467 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755459 4988 scope.go:122] "RemoveContainer" containerID="c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913" Oct 30 21:49:33.755703 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755682 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913"} err="failed to get container status \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": rpc error: code = NotFound desc = could not find container \"c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913\": container with ID starting with c730e96d8fb1d6abc743305f40c52bad5a6b8e7eec653e3a530ba7d05ad90913 not found: ID does not exist" Oct 30 21:49:33.755754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755703 4988 scope.go:122] "RemoveContainer" containerID="a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473" Oct 30 21:49:33.755932 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755899 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473"} err="failed to get container status \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": rpc error: code = NotFound desc = could not find container \"a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473\": container with ID starting with a2a1a02786310de5b550263cd955e2b6d1e99504fd37cc89e19324efd2ea9473 not found: ID does not exist" Oct 30 21:49:33.755981 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.755934 4988 scope.go:122] "RemoveContainer" containerID="507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8" Oct 30 21:49:33.756124 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.756109 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8"} err="failed to get container status \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": rpc error: code = NotFound desc = could not find container \"507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8\": container with ID starting with 507b3876092f399cddbf3274b4eb6de2eccf1690f481e5636a2cfb1ffc5928a8 not found: ID does not exist" Oct 30 21:49:33.756170 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.756125 4988 scope.go:122] "RemoveContainer" containerID="c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" Oct 30 21:49:33.764196 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.764175 4988 scope.go:122] "RemoveContainer" containerID="650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" Oct 30 21:49:33.772717 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.772687 4988 scope.go:122] "RemoveContainer" containerID="04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" Oct 30 21:49:33.780934 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.780913 4988 scope.go:122] "RemoveContainer" containerID="c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" Oct 30 21:49:33.781216 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:33.781195 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": container with ID starting with c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89 not found: ID does not exist" containerID="c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" Oct 30 21:49:33.781294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.781230 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89"} err="failed to get container status \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": rpc error: code = NotFound desc = could not find container \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": container with ID starting with c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89 not found: ID does not exist" Oct 30 21:49:33.781294 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.781254 4988 scope.go:122] "RemoveContainer" containerID="650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" Oct 30 21:49:33.781649 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:33.781630 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": container with ID starting with 650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0 not found: ID does not exist" containerID="650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" Oct 30 21:49:33.781706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.781656 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0"} err="failed to get container status \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": rpc error: code = NotFound desc = could not find container \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": container with ID starting with 650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0 not found: ID does not exist" Oct 30 21:49:33.781706 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.781671 4988 scope.go:122] "RemoveContainer" containerID="04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" Oct 30 21:49:33.781890 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:33.781874 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": container with ID starting with 04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079 not found: ID does not exist" containerID="04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" Oct 30 21:49:33.781942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.781894 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079"} err="failed to get container status \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": rpc error: code = NotFound desc = could not find container \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": container with ID starting with 04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079 not found: ID does not exist" Oct 30 21:49:33.781942 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.781921 4988 scope.go:122] "RemoveContainer" containerID="c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" Oct 30 21:49:33.782143 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782126 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89"} err="failed to get container status \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": rpc error: code = NotFound desc = could not find container \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": container with ID starting with c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89 not found: ID does not exist" Oct 30 21:49:33.782188 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782143 4988 scope.go:122] "RemoveContainer" containerID="650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" Oct 30 21:49:33.782368 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782351 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0"} err="failed to get container status \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": rpc error: code = NotFound desc = could not find container \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": container with ID starting with 650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0 not found: ID does not exist" Oct 30 21:49:33.782433 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782368 4988 scope.go:122] "RemoveContainer" containerID="04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" Oct 30 21:49:33.782609 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782591 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079"} err="failed to get container status \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": rpc error: code = NotFound desc = could not find container \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": container with ID starting with 04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079 not found: ID does not exist" Oct 30 21:49:33.782664 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782610 4988 scope.go:122] "RemoveContainer" containerID="c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89" Oct 30 21:49:33.782848 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782827 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89"} err="failed to get container status \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": rpc error: code = NotFound desc = could not find container \"c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89\": container with ID starting with c5a0620ad80512d9f0bf4412edc1de78ec6cddbf1a9cb8788bee83f71e2fad89 not found: ID does not exist" Oct 30 21:49:33.782901 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.782848 4988 scope.go:122] "RemoveContainer" containerID="650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0" Oct 30 21:49:33.783066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.783040 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0"} err="failed to get container status \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": rpc error: code = NotFound desc = could not find container \"650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0\": container with ID starting with 650456e3f4902ba7d364818a171a61460153b7a99aec3c6fdfc93eb0e45d3fd0 not found: ID does not exist" Oct 30 21:49:33.783066 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.783064 4988 scope.go:122] "RemoveContainer" containerID="04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079" Oct 30 21:49:33.783316 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.783298 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079"} err="failed to get container status \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": rpc error: code = NotFound desc = could not find container \"04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079\": container with ID starting with 04208bc4495786ab9f32b37c4f8617ba9570f8da5505ebe49830a07851b87079 not found: ID does not exist" Oct 30 21:49:33.890118 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.890031 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7"] Oct 30 21:49:33.895344 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:33.895318 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-77657866bd-2wmc7"] Oct 30 21:49:34.004442 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.004391 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t"] Oct 30 21:49:34.015663 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.015629 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" path="/var/lib/kubelet/pods/06693e03-b579-4de8-b9cf-99f2cfae2a46/volumes" Oct 30 21:49:34.016018 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.016003 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-66dfd66cdb-wzj9t"] Oct 30 21:49:34.209140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.209065 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.354109 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.354075 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.526092 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.525995 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:49:34.526298 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.526272 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="alertmanager" containerID="cri-o://04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168" gracePeriod=120 Oct 30 21:49:34.526373 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.526346 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="config-reloader" containerID="cri-o://e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6" gracePeriod=120 Oct 30 21:49:34.535255 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.535212 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:49:34.535568 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.535539 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="prometheus" containerID="cri-o://41ef28da10cb47aac6e9453c1b1038bde2a3e6044c8f9c43654bf90f8f69985c" gracePeriod=600 Oct 30 21:49:34.535646 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.535562 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="config-reloader" containerID="cri-o://7f65bf2aaee4b02210120b7c1d218b493efac85b9fb28ad20806033248cd64df" gracePeriod=600 Oct 30 21:49:34.535689 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.535561 4988 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="thanos-sidecar" containerID="cri-o://177b906f10655232c69c1d0d60ed603e006d11c2a2f09b67d13fd766ba82d0ca" gracePeriod=600 Oct 30 21:49:34.544051 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.544010 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.560321 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.560265 4988 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="alertmanager" probeResult="failure" output="Get \"http://10.132.0.160:9093/-/ready\": dial tcp 10.132.0.160:9093: connect: connection refused" Oct 30 21:49:34.695633 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.695576 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.736256 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.736213 4988 generic.go:356] "Generic (PLEG): container finished" podID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerID="e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6" exitCode=0 Oct 30 21:49:34.736458 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.736277 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerDied","Data":"e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6"} Oct 30 21:49:34.742650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.742616 4988 generic.go:356] "Generic (PLEG): container finished" podID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerID="177b906f10655232c69c1d0d60ed603e006d11c2a2f09b67d13fd766ba82d0ca" exitCode=0 Oct 30 21:49:34.742650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.742647 4988 generic.go:356] "Generic (PLEG): container finished" podID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerID="7f65bf2aaee4b02210120b7c1d218b493efac85b9fb28ad20806033248cd64df" exitCode=0 Oct 30 21:49:34.742650 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.742656 4988 generic.go:356] "Generic (PLEG): container finished" podID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerID="41ef28da10cb47aac6e9453c1b1038bde2a3e6044c8f9c43654bf90f8f69985c" exitCode=0 Oct 30 21:49:34.742943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.742686 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerDied","Data":"177b906f10655232c69c1d0d60ed603e006d11c2a2f09b67d13fd766ba82d0ca"} Oct 30 21:49:34.742943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.742749 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerDied","Data":"7f65bf2aaee4b02210120b7c1d218b493efac85b9fb28ad20806033248cd64df"} Oct 30 21:49:34.742943 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.742778 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerDied","Data":"41ef28da10cb47aac6e9453c1b1038bde2a3e6044c8f9c43654bf90f8f69985c"} Oct 30 21:49:34.793109 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.793077 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:49:34.841955 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.841916 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.848593 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848567 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848617 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848644 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848670 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848721 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848696 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848729 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848819 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848858 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848886 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.848941 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848920 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.849208 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.848959 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.849208 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.849011 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.849208 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.849045 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config\") pod \"bc1f9485-e328-43c5-92a0-d1fd090f846c\" (UID: \"bc1f9485-e328-43c5-92a0-d1fd090f846c\") " Oct 30 21:49:34.849208 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.849099 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:34.849208 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.849110 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:34.849489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.849336 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.849489 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.849359 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.850437 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.850371 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:34.851928 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.851893 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "kube-api-access-8sqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:34.853249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.852265 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:34.853249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.852510 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:34.853249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.852629 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:34.853249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.852788 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:34.853677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.853249 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:34.853677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.853364 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:34.853677 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.853387 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:34.855040 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.855019 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 21:49:34.865744 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.865705 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config" pod "bc1f9485-e328-43c5-92a0-d1fd090f846c" (UID: "bc1f9485-e328-43c5-92a0-d1fd090f846c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:34.950799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950763 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqcn\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-kube-api-access-8sqcn\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.950799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950793 4988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-secret-prometheus-web-tls-ca\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.950799 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950805 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950815 4988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950850 4988 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" " Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950861 4988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950870 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1f9485-e328-43c5-92a0-d1fd090f846c-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950878 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1f9485-e328-43c5-92a0-d1fd090f846c-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950889 4988 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950898 4988 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1f9485-e328-43c5-92a0-d1fd090f846c-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.951138 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.950911 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1f9485-e328-43c5-92a0-d1fd090f846c-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:34.963027 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:34.962999 4988 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba") on node "ip-10-0-141-115.us-west-1.compute.internal" Oct 30 21:49:35.052325 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.052227 4988 reconciler_common.go:299] "Volume detached for volume \"pvc-fb65aa63-2762-435b-a8ef-8892b9ffe1f6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0913d5896cba88aba\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"csi-6011946b6b6cb264a05ab54b08c25f594cd178d9fdfeba56ee1265902d17ce31\"" Oct 30 21:49:35.380113 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.380075 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:49:35.455785 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.455743 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.455966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.455827 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.455966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.455886 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.455966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.455931 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.455966 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.455960 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.456183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.455993 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.456183 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.456021 4988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume\") pod \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\" (UID: \"3b9be458-1a6c-4bb9-a7b5-36f18adcb228\") " Oct 30 21:49:35.456387 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.456353 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:35.458135 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.458101 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:35.458358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.458334 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:35.458512 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.458494 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "kube-api-access-n99wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:35.458747 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.458719 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:35.462461 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.462422 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:35.469433 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.469386 4988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config" pod "3b9be458-1a6c-4bb9-a7b5-36f18adcb228" (UID: "3b9be458-1a6c-4bb9-a7b5-36f18adcb228"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:35.556619 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556574 4988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n99wq\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-kube-api-access-n99wq\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.556619 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556612 4988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-tls-assets\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.556842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556644 4988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-web-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.556842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556655 4988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.556842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556665 4988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-out\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.556842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556674 4988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-cluster-tls-config\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.556842 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.556683 4988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3b9be458-1a6c-4bb9-a7b5-36f18adcb228-config-volume\") on node \"ip-10-0-141-115.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:35.751128 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.751088 4988 generic.go:356] "Generic (PLEG): container finished" podID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerID="04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168" exitCode=0 Oct 30 21:49:35.751353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.751159 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 21:49:35.751353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.751165 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerDied","Data":"04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168"} Oct 30 21:49:35.751353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.751250 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3b9be458-1a6c-4bb9-a7b5-36f18adcb228","Type":"ContainerDied","Data":"a9aebdeaf43c37b4b2b0bfa17d9affb7d2e5d3a9eb7f961d380c143fcafef0cd"} Oct 30 21:49:35.751353 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.751280 4988 scope.go:122] "RemoveContainer" containerID="e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6" Oct 30 21:49:35.755172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.753952 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"bc1f9485-e328-43c5-92a0-d1fd090f846c","Type":"ContainerDied","Data":"074b771c965ecd63e0f60fadcbe4281333602ea009ca8a325294696bedf1a92a"} Oct 30 21:49:35.755172 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.754012 4988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 21:49:35.762109 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.762083 4988 scope.go:122] "RemoveContainer" containerID="04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168" Oct 30 21:49:35.770959 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.770935 4988 scope.go:122] "RemoveContainer" containerID="415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8" Oct 30 21:49:35.779360 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.779338 4988 scope.go:122] "RemoveContainer" containerID="e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6" Oct 30 21:49:35.779723 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:35.779701 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6\": container with ID starting with e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6 not found: ID does not exist" containerID="e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6" Oct 30 21:49:35.779780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.779733 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6"} err="failed to get container status \"e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6\": rpc error: code = NotFound desc = could not find container \"e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6\": container with ID starting with e73a90000cb78c17de5e9ced74d9f850105098aab0c664e8598c14aa22b250e6 not found: ID does not exist" Oct 30 21:49:35.779780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.779754 4988 scope.go:122] "RemoveContainer" containerID="04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168" Oct 30 21:49:35.780002 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:35.779986 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168\": container with ID starting with 04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168 not found: ID does not exist" containerID="04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168" Oct 30 21:49:35.780049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.780006 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168"} err="failed to get container status \"04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168\": rpc error: code = NotFound desc = could not find container \"04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168\": container with ID starting with 04ae10bd46d3720319de6b13b28c1c63bba3d688808199f203ef8b57eb2d2168 not found: ID does not exist" Oct 30 21:49:35.780049 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.780018 4988 scope.go:122] "RemoveContainer" containerID="415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8" Oct 30 21:49:35.780227 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:35.780209 4988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8\": container with ID starting with 415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8 not found: ID does not exist" containerID="415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8" Oct 30 21:49:35.780307 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.780239 4988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8"} err="failed to get container status \"415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8\": rpc error: code = NotFound desc = could not find container \"415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8\": container with ID starting with 415c13d0e211a7c710470214cc6b74bf32cf1df9915ead8ae924c866d76327a8 not found: ID does not exist" Oct 30 21:49:35.780307 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.780263 4988 scope.go:122] "RemoveContainer" containerID="177b906f10655232c69c1d0d60ed603e006d11c2a2f09b67d13fd766ba82d0ca" Oct 30 21:49:35.788780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.788754 4988 scope.go:122] "RemoveContainer" containerID="7f65bf2aaee4b02210120b7c1d218b493efac85b9fb28ad20806033248cd64df" Oct 30 21:49:35.797811 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.797788 4988 scope.go:122] "RemoveContainer" containerID="41ef28da10cb47aac6e9453c1b1038bde2a3e6044c8f9c43654bf90f8f69985c" Oct 30 21:49:35.806946 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.806924 4988 scope.go:122] "RemoveContainer" containerID="edc868e8a2fe6cc31a91f6ebcb95ab49eb47927a80eeaf465ea60f532ed99eba" Oct 30 21:49:35.917992 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.917943 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:49:35.967179 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:35.967135 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 21:49:36.016835 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:36.016754 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" path="/var/lib/kubelet/pods/34d5e605-e3b7-43f8-9573-d93071ac5b93/volumes" Oct 30 21:49:36.017362 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:36.017344 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" path="/var/lib/kubelet/pods/bc1f9485-e328-43c5-92a0-d1fd090f846c/volumes" Oct 30 21:49:36.077845 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:36.077806 4988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:49:36.082358 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:36.082326 4988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 21:49:36.867171 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:36.867131 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.011822 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:37.011783 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.189157 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:37.189117 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.016388 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:38.016353 4988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" path="/var/lib/kubelet/pods/3b9be458-1a6c-4bb9-a7b5-36f18adcb228/volumes" Oct 30 21:49:38.495792 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:38.495755 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.644843 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:38.644798 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.793430 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:38.793316 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.943786 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:38.943751 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.235899 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:42.235850 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.406265 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:42.406225 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:49.937342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:49.937305 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.085422 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:50.085367 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.396448 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:49:56.396393 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:49:56.839997 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:49:56.839900 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.086716 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:03.086678 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.261636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:03.261601 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.462508 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:09.462466 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:10.472636 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:10.472596 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:10.671610 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:10.671554 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.043778 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:12.043739 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.238844 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:12.238803 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:19.676102 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:19.676065 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2kjtc_6ed71fea-5f6c-4ba4-be09-b524c1abfc4a/global-pull-secret-syncer/0.log" Oct 30 21:50:21.839651 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:21.839614 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ldcc4_70d94d6f-863f-4819-b36f-5d2c753471bd/connection-checker/0.log" Oct 30 21:50:22.350986 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:22.350954 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-sg6jp_470aba85-18f4-4acc-9e98-6233c2ae3c9c/connection-checker/0.log" Oct 30 21:50:22.786522 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:22.786494 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tkhfn_cacd9cad-031b-4a85-a86d-a04eecb18d95/connection-checker/0.log" Oct 30 21:50:23.272771 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:23.272738 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" Oct 30 21:50:24.815057 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:24.815020 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-115.us-west-1.compute.internal_124b8400391b2921b52ea70b658a260b/haproxy/0.log" Oct 30 21:50:26.395434 ip-10-0-141-115 kubenswrapper[4988]: E1030 21:50:26.395378 4988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdlwf" containerName="extractor" Oct 30 21:50:26.872726 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:26.872639 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.466297 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466255 4988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk"] Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466602 4988 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="prometheus" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466614 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="prometheus" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466623 4988 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="init-config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466629 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="init-config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466637 4988 cpu_manager.go:414] "containerMap: removing container" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="prom-label-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466642 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="prom-label-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466652 4988 cpu_manager.go:414] "containerMap: removing container" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="alertmanager" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466658 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="alertmanager" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466672 4988 cpu_manager.go:414] "containerMap: removing container" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="prom-label-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466676 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="prom-label-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466691 4988 cpu_manager.go:414] "containerMap: removing container" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="kube-rbac-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466696 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="kube-rbac-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466701 4988 cpu_manager.go:414] "containerMap: removing container" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="method-gate" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466708 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="method-gate" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466715 4988 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="thanos-sidecar" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466720 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="thanos-sidecar" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466728 4988 cpu_manager.go:414] "containerMap: removing container" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="method-gate" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466733 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="method-gate" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466740 4988 cpu_manager.go:414] "containerMap: removing container" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="init-config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466745 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="init-config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466752 4988 cpu_manager.go:414] "containerMap: removing container" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466756 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466763 4988 cpu_manager.go:414] "containerMap: removing container" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466767 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="config-reloader" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466775 4988 cpu_manager.go:414] "containerMap: removing container" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="kube-rbac-proxy" Oct 30 21:50:29.466777 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466780 4988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="kube-rbac-proxy" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466832 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="prom-label-proxy" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466842 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="kube-rbac-proxy" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466848 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="method-gate" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466854 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="thanos-sidecar" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466860 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d5e605-e3b7-43f8-9573-d93071ac5b93" containerName="method-gate" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466866 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="config-reloader" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466871 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="config-reloader" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466876 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="kube-rbac-proxy" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466882 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1f9485-e328-43c5-92a0-d1fd090f846c" containerName="prometheus" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466886 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="06693e03-b579-4de8-b9cf-99f2cfae2a46" containerName="prom-label-proxy" Oct 30 21:50:29.467727 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.466892 4988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9be458-1a6c-4bb9-a7b5-36f18adcb228" containerName="alertmanager" Oct 30 21:50:29.469979 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.469948 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.502511 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.502466 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-lib-modules\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.502696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.502563 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-podres\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.502696 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.502642 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-proc\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.502773 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.502695 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-sys\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.502773 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.502724 4988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x29z\" (UniqueName: \"kubernetes.io/projected/f830adf3-6c0b-4783-b490-ea35c0bc1257-kube-api-access-7x29z\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.527910 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.527872 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk"] Oct 30 21:50:29.576110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.576065 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dfcks\"/\"kube-root-ca.crt\"" Oct 30 21:50:29.576110 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.576100 4988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dfcks\"/\"default-dockercfg-k474x\"" Oct 30 21:50:29.576342 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.576138 4988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dfcks\"/\"openshift-service-ca.crt\"" Oct 30 21:50:29.603232 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603191 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-proc\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603232 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603231 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-sys\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603257 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x29z\" (UniqueName: \"kubernetes.io/projected/f830adf3-6c0b-4783-b490-ea35c0bc1257-kube-api-access-7x29z\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603300 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-lib-modules\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603331 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-sys\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603331 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-proc\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603382 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-podres\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603332 4988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-podres\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.603506 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.603442 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f830adf3-6c0b-4783-b490-ea35c0bc1257-lib-modules\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.667212 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.667179 4988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x29z\" (UniqueName: \"kubernetes.io/projected/f830adf3-6c0b-4783-b490-ea35c0bc1257-kube-api-access-7x29z\") pod \"perf-node-gather-daemonset-jmkxk\" (UID: \"f830adf3-6c0b-4783-b490-ea35c0bc1257\") " pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:29.782732 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:29.782633 4988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:30.173597 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:30.173562 4988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk"] Oct 30 21:50:30.176708 ip-10-0-141-115 kubenswrapper[4988]: W1030 21:50:30.176673 4988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf830adf3_6c0b_4783_b490_ea35c0bc1257.slice/crio-d9813254b31cd7c3cc7d096037885dde965a3aee3f661593e3234c4ce143feb4 WatchSource:0}: Error finding container d9813254b31cd7c3cc7d096037885dde965a3aee3f661593e3234c4ce143feb4: Status 404 returned error can't find the container with id d9813254b31cd7c3cc7d096037885dde965a3aee3f661593e3234c4ce143feb4 Oct 30 21:50:30.999754 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:30.999714 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" event={"ID":"f830adf3-6c0b-4783-b490-ea35c0bc1257","Type":"ContainerStarted","Data":"1ea8b9368914ffbf89892e3ebec3303c7dc38fca3f7a899b2d58910dd4e3bd9b"} Oct 30 21:50:31.000182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:30.999769 4988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" event={"ID":"f830adf3-6c0b-4783-b490-ea35c0bc1257","Type":"ContainerStarted","Data":"d9813254b31cd7c3cc7d096037885dde965a3aee3f661593e3234c4ce143feb4"} Oct 30 21:50:31.000182 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:30.999926 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:31.116527 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:31.116465 4988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" podStartSLOduration=2.116447087 podStartE2EDuration="2.116447087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:31.11565031 +0000 UTC m=+4459.223146224" watchObservedRunningTime="2026-10-30 21:50:31.116447087 +0000 UTC m=+4459.223942992" Oct 30 21:50:31.320249 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:31.320167 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-tvpjf_7ea62097-316d-49f5-bccf-dba48029bb6d/cluster-monitoring-operator/0.log" Oct 30 21:50:31.567050 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:31.567006 4988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l2csm_975f6dec-89de-4c6e-ab23-a665d5d76ced/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.249780 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:35.249739 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9zg62_4538254f-130c-4b0a-b8c4-2fc095d0758d/node-exporter/0.log" Oct 30 21:50:35.668471 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:35.668438 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9zg62_4538254f-130c-4b0a-b8c4-2fc095d0758d/kube-rbac-proxy/0.log" Oct 30 21:50:36.090663 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:36.090594 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9zg62_4538254f-130c-4b0a-b8c4-2fc095d0758d/init-textfile/0.log" Oct 30 21:50:37.014140 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:37.014106 4988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dfcks/perf-node-gather-daemonset-jmkxk" Oct 30 21:50:49.763994 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:49.763962 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zbpwc_db8922f9-31fe-4afd-b227-313eecf0fcaa/networking-console-plugin/0.log" Oct 30 21:50:52.459058 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:52.459029 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/1.log" Oct 30 21:50:52.467678 ip-10-0-141-115 kubenswrapper[4988]: I1030 21:50:52.467641 4988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2lsbc_3d68f309-a728-4000-a631-86ef2c50b3e0/console-operator/2.log"